Measure for Measure(ment)
James ShanteauJames Shanteau
University Distinguished Professor of University Distinguished Professor of Psychology Emeritus Psychology Emeritus
Kansas State UniversityKansas State University
Bayes Conference – 2/14
Goals and Caveats
Three goals:(1) Summarize why measurement is essential(2) Review some prior studies using measurement(3) Consider what more needs to be done
Goals and Caveats
Three goals:(1) Summarize why measurement is essential(2) Review some prior studies using measurement(3) Consider what more needs to be doneCaveats:(1) Concern with psychological measurement of
stim-ulus values, sometimes called psychological scaling
(2) Focus on graphical analyses, which are based on underlying numerical estimates of psychological measures
Quotes About Measurement“Thou shall not have in thine house diverse measures.
But a perfect & just measure shalt thou have” (Deuteronomy)
“If you cannot measure it then it is not science” (Lord Kelvin)
“The first steps in the path of discovery … the first approximate measures, are those which add most to the existing knowledge of mankind” (Charles Babbage)
“To understand God’s thoughts one must study statistics … the measure of his purpose” (Florence Nightingale)
“Whenever you can, count” (Sir Francis Galton)
“Research is 4 things: brains with which to think, eyes with which to see, machines with which to measure, & 4th money” m (Albert Szent-Gyorgyi)
Quote by Psychologists on Measurement
“In order to formulate quantitative laws (in psychology),
the relevant properties must be expressible by numbers.
The process by which scientists (do this) is called
measure-ment.” (Coombs, Dawes, & Tversky, 1970)
Where Is Discussion of Measurement? To find additional comments on Measurement in
Psychology, I looked at recent books with “Measurement” in title, eg:
Edwards, W., (Ed.) (1992). “Utility theories: Measurements and applications.” Boston: Kluwer Academic Press.
Weiss, D. J. (2006) “Analysis of variance and functional measurement.” Oxford: Oxford University Press.
Lockhart, P. (2012). “Measurement.” Cambridge, MA: Belknap Press of Harvard University Press.
Only 1 of 3 included any content on psych measurementGuess which one?
Where Is Discussion of Measurement? To find additional comments on Measurement in
Psychology, I looked at recent books with “Measurement” in title, eg:
Edwards, W., (Ed.) (1992). “Utility theories: Measurements and applications.” Boston: Kluwer Academic Press.
Weiss, D. J. (2006) “Analysis of variance and functional measurement.” Oxford: Oxford University Press.
Lockhart, P. (2012). “Measurement.” Cambridge, MA: Belknap Press of Harvard University Press.
Only 1 of 3 included any content on psych measurementGuess which one?Only Weiss book includes discussion of actual
measurementsThe Dilemma: Why is “Measurement” widely talked
about, but so rarely done?
Some Preliminary Comments
(1)(1) Measures = parameters of models, stated or Measures = parameters of models, stated or
unstatedunstated
=> Validity of measures depends on validity of models=> Validity of measures depends on validity of models
(2) Estimating Scaling values ≠ Weighting values(2) Estimating Scaling values ≠ Weighting values
=> Different estimation methods required=> Different estimation methods required
(3) Typically try to get interval-scale estimates(3) Typically try to get interval-scale estimates
=> Need to establish 2 points, eg, slope & intercept=> Need to establish 2 points, eg, slope & intercept
(4) Computer programs simplify getting actual (4) Computer programs simplify getting actual
measuresmeasures
=> => “Its too hard to do measurement” “Its too hard to do measurement” is no longer an is no longer an
excuseexcuse
Example of Weight MeasurementMy 1My 1stst study as grad student involved estimation study as grad student involved estimation of serial position weights in dynamic decision of serial position weights in dynamic decision making tasks, eg, probability revisionmaking tasks, eg, probability revision
These curves show weights of serial positions for These curves show weights of serial positions for different lengths of stimuli, Resp 4 to Resp 15different lengths of stimuli, Resp 4 to Resp 15
See generalized recencySee generalized recency
Note: weight estimatesNote: weight estimates
are not normalizedare not normalized
Example of Weight Measurement
How to Compare Weights?
When Are Weight Measures Important?
In study of expertise by Ettenson, Krogstad, & In study of expertise by Ettenson, Krogstad, & Shanteau, 3 groups of auditors judged common Shanteau, 3 groups of auditors judged common set of cases described by 8 cuesset of cases described by 8 cues
Expert (Partners) & intermediate (Seniors) Expert (Partners) & intermediate (Seniors) auditors relied primarily on 1 cue, whereas auditors relied primarily on 1 cue, whereas graduate (Stu-dents) made use of many more graduate (Stu-dents) made use of many more cuescues
Has implications for teachingHas implications for teaching
students to focus on whatstudents to focus on what
is vital in a given contextis vital in a given context
Example of Scale Value Measurement
Why Care About Measurement of Scale Values?
In an analysis of willingness to donate various In an analysis of willingness to donate various organs while living, Skowronski & Shanteau had organs while living, Skowronski & Shanteau had 4 groups evaluate various donations while living4 groups evaluate various donations while living
Found all groups (even Anti-Donors) were willing Found all groups (even Anti-Donors) were willing to donate to relativesto donate to relatives
This finding has important implications for This finding has important implications for increasing rates of organ donationincreasing rates of organ donation
Everyone willing to donateEveryone willing to donate… … under some situationsunder some situations
No one is No one is Anti-DonationAnti-Donation
But most are But most are Anti-Stranger Anti-Stranger
Example of Joint Weight & Scale Value Measurements
My contribution to measurements involved using My contribution to measurements involved using bilinear models to estimate both weight & scale bilinear models to estimate both weight & scale values; bilinearity => diverging fan of straight values; bilinearity => diverging fan of straight lineslines
On left, objective values => irregular pattern On left, objective values => irregular pattern unclearunclear
On right, subjective values => bilinear pattern On right, subjective values => bilinear pattern evidentevident
Shows linkage between model Shows linkage between model testing & measurementtesting & measurement
See support for SEU model:See support for SEU model: SEU = Subj Prob x UtilitySEU = Subj Prob x Utility
What About Non-Numeric, Verbal Stimuli?
Traditionally measurement approaches Traditionally measurement approaches transform numeric stimuli into psych values, eg, transform numeric stimuli into psych values, eg, psychophysicspsychophysics
But measurement not constrained to numeric But measurement not constrained to numeric stimulistimuli
Challenge is getting interval scale valuesChallenge is getting interval scale values
This can be solved by using This can be solved by using
anchor stimuli, eg,anchor stimuli, eg,
““No Chance” No Chance” & & “Sure Thing”“Sure Thing”
““$.50” $.50” & & “$60”“$60”
Other Examples of Measurement
More Quotes About Measurement
““If measurement matters at all, it is because it If measurement matters at all, it is because it
must have some conceivable effect on must have some conceivable effect on
decisions and behaviour” decisions and behaviour” (Douglas W. Hubbard)(Douglas W. Hubbard)
“ ’“ ’By measurement to knowledge’ I should write By measurement to knowledge’ I should write
as a motto above the entrance to every labora-as a motto above the entrance to every labora-
tory” tory” (Heike Kamerlingh Onnes)(Heike Kamerlingh Onnes)
““An experiment is a question which science An experiment is a question which science
poses to nature and a measurement is the poses to nature and a measurement is the
recording of nature’s answer” recording of nature’s answer” (Max Planck)(Max Planck)
What’s Holding Back Measurement?Given benefits of measurement, why isn’t
everyone estimating subjective values?Three possible reasons:(1) Confusions over producing interval scales
=> Need more examples to illustrate “how to do”
(2) Seemingly arbitrary assumption required=> That is why measurement & model testing linked
(3) “Forrest Young Problem” => He said MDS too sophisticated for most users=> Need to make measurement easy
Some Caveats About Measurement
““Those who think ‘Science is Measurement’ Those who think ‘Science is Measurement’
should search Darwin’s works for numbers and should search Darwin’s works for numbers and
equations” equations” (David Hubbel)(David Hubbel)
““Our clocks do not measure time … time is Our clocks do not measure time … time is
defined to be what our clocks measure” defined to be what our clocks measure”
(Anonymous, NIST)(Anonymous, NIST)
““Not everything that counts can be counted and Not everything that counts can be counted and
not everything that can be counted counts” not everything that can be counted counts”
(Albert Einstein) (Albert Einstein)
Comments and Questions? Your turn …
Norman Anderson
Measure for Measure(ment)
James ShanteauJames Shanteau
University Distinguished Professor of University Distinguished Professor of Psychology Emeritus Psychology Emeritus
Kansas State UniversityKansas State University
Bayes Conference – 2/14
Issue 3: What About Individual Differences?
Group analyses often conceal sizeable individual Group analyses often conceal sizeable individual differencesdifferences
One advantage of FM is ability to analyze both One advantage of FM is ability to analyze both levelslevels
See differences in both model shape & valuesSee differences in both model shape & values
Norman Anderson on Measurement “The logic of (FM) consists in using the postulated
behavior-al laws to induce a scaling on the dependent variable” (1962)
“A guiding principle of FM is that measurement scales are derivative from substantive theory” (1970)
“Measurement is fundamental” (1996)“FM reverses the traditional approach & makes
measurement an organic component of substantive investigation” (1981)
“All that is necessary is to have a valid integration function. That function provides base & frame for measurement” (1981)
“Measurement… is the link between the world of behavior & the world of science … (it) is a vital feature of science … meas-urement is thus an integral part of substantive theory” (2001)
“Much of progress consists in improvements in measurement, both in its empirical base & its conceptual base” (2001)
Norman Anderson (NHA) Story 50 years ago, NHA was debating with advocates of
Conjoint Measurement (CM). One of his major arguments was that CM wasn’t being used to actually measure anything, but FM was.
However, there were almost no examples of actual measures by FM at that time.
NHA quickly put together a paper with actual measures of adjectives => it was published in Psych Reports (?) => I have been unable to find either the paper or the citation
However, my memory of this story was the motivation for the present talk, ie, we talk-the-talk about measurement, but how often do we walk-the-walk?
If measurement is such a vital part of FM, why don’t we see measured values reported in most every study
But, such reports of numerical measures is rare. Why?
Purpose & OrganizationPurposePurpose:: To revisit role that measurement plays To revisit role that measurement plays
in Functional Measurement (FM)in Functional Measurement (FM)
Organization of TalkOrganization of Talk::
(1)(1) Describe role of measurement in FM logicDescribe role of measurement in FM logic
(2)(2) Present examples of measurement in FM Present examples of measurement in FM studies studies
(3)(3) Consider some possible barriers to use of FMConsider some possible barriers to use of FM
(4)(4) Take comments & answer questions (if I can)Take comments & answer questions (if I can)
Example of Joint Weight & Scale Value Measurements
Can see application of bilinear analysis in Can see application of bilinear analysis in reanalysis of Tversky (left panel)reanalysis of Tversky (left panel)
See similar patterns in Slovic & Lichenstein dataSee similar patterns in Slovic & Lichenstein dataNote: both original analyses involved additivity Note: both original analyses involved additivity (par-allelism) assumptions based on objective (par-allelism) assumptions based on objective valuesvalues
Issue 3: What About Individual Differences?
Group analyses often conceal sizeable individual Group analyses often conceal sizeable individual differencesdifferences
One advantage of FM is ability to analyze both One advantage of FM is ability to analyze both levelslevels
Example of FM Using RTsNearly all studies using FM have used rating dataNearly all studies using FM have used rating dataOne exception: Weiss’ work using ordinal dataOne exception: Weiss’ work using ordinal dataAnother exception: FM analysis of RTs in Another exception: FM analysis of RTs in question-answering task introduced by question-answering task introduced by AndersonAnderson
Results provided new insights into priming Results provided new insights into priming effectseffects
Key is bilinear analysis of cumulative Response Key is bilinear analysis of cumulative Response TimeTime
Allows bilinear scalingAllows bilinear scaling∴ ∴ Can “measure” timesCan “measure” times=> just like rating data=> just like rating data
Priming found in both mem-Priming found in both mem-ory & decision timesory & decision times
Decision priming over 2xDecision priming over 2xeffect of memory priming effect of memory priming
More Quotes on Measurement “Measurement is the 1st step to control & eventually
to improvement. If you can’t measure something, you can’t understand it” (H. James Herrington)
If measurement matters at all, it is because it must have some conceivable effect on decisions & behaviour” (Douglas W. Hubbard)
“’By measurement to knowledge’ I should write as a motto above entrance to every laboratory” (Heike Kamerlingh Onnes)
“An experiment is a question which science poses to nature, & a measurement is the recording of nature’s answer” (Max Planck)
“If you cannot measure it you cannot control it” (J Grebe)
“Man is the measure of all things” (Protagoras)
Quote by Psychologists on Measurement “Although measurement & scaling have sometimes
been used interchangeably in the literature, we have
chosen to distin-guish between them. Measurement is
concerned with the con-ditions under which various types
of scales can be construct-ed. The actual process of
assigning numbers … is called scal-ing” (Coombs, 1964)
“In order to formulate quantitative laws (in psychology),
the relevant properties must be expressible by numbers.
The process by which scientists (do this) is called
measure-ment.” (Coombs, Dawes, & Tversky, 1970)
Some Preliminary Comments
(1)(1) Measures = parameters of models, stated or Measures = parameters of models, stated or
unstatedunstated
=> Validity of measures depends on validity of models=> Validity of measures depends on validity of models
=> There is direct link between model fit & => There is direct link between model fit &
measurementmeasurement
(2) Estimating Scaling values ≠ Weighting values(2) Estimating Scaling values ≠ Weighting values
=> Different estimation methods required=> Different estimation methods required
=> Lead to different interpretation and usage=> Lead to different interpretation and usage
(3) Typically try to get interval-scale estimates(3) Typically try to get interval-scale estimates
=> Need to establish 2 points, eg, slope & intercept=> Need to establish 2 points, eg, slope & intercept
(4) Computer programs simplify getting actual (4) Computer programs simplify getting actual
measuresmeasures
=> “Its hard to do measurement” is no longer an => “Its hard to do measurement” is no longer an
excuseexcuse
Some Caveats on Measurement“Not everything that is observable & measurable is
predictable, no matter how complete our past observations may have been” (Sir William McCrea)
“Those who think ‘Science is Measurement’ should search Darwin’s works for numbers & equations” (David Hubel)
“Investigators seem to have settled for what is measur-able instead of measuring what they would really like to know” (E D Pellegrino) => Note: Drunk & Lamppost problem
“Our clocks do not measure time … Time is defined to be what our clocks measure” (Anonymous, NIST)
“Not everything that counts can be counted & not every-thing that can be counted counts” (Albert Einstein)
Are Measured Values Constant Across Tasks?
Within many tasks, eg, impression formation, Within many tasks, eg, impression formation, mea-sured values appear to be more-or-less mea-sured values appear to be more-or-less constantconstant
But across tasks, see sizable differences, eg, in But across tasks, see sizable differences, eg, in sub-jective probability values for gambles vs sub-jective probability values for gambles vs datingdating