thinking with your body and other things
TRANSCRIPT
Thinking with your Body
and Other Things
David Kirsh
Cognitive Science
UCSD
HOW DO WE
MOVE THOUGHT FORWARD?
Hand drags the eye - pacer
How do we use our …
Eyes
Hands
Bodies
Tools
Nearby objects
To help us think?
Thinking with our eyes
Will this design work?
Which square becomes the cube’s top?
X D
C
B
A F
E
D is the top square
D D
Mental Simulation
D C
B A F
E
Some thinking involves
mental simulation while looking
Projection: another element of
visual thinking
See things that aren’t there
Move 11 to where 15 is
Initially visual search to locate 11 and 15
1 1
1 5
We project faint mental images onto external or internal visual anchors or indices (FINSTs)
11 to 15
How many moves can you reliably project?
1
1 1
Now 1 to 11
Visual search
Projection
Manipulation then Create change
Project
Project Create Interactive strategy
THESIS
Thinking often involves
making models or simulations
through ‘active’ perception,
projection and manipulation.
Thinking often involves
manipulating external
structures (as well as internal)
projections.
Agenda
Main Theoretical Claims
Thinking through the Body
Study of Dance Making
Thinking through Instruments
Action vs. Observation
Thinking through Ordinary Objects,
Bottles, Representations, …
Agenda
Main Theoretical Claims
Thinking through the Body
Study of Dance Making
Thinking through Instruments
Action vs. Observation
Thinking through Ordinary Objects,
Bottles, Representations, …
ONE: THINKING IS OFTEN
INTERACTIVE
Claim
How do we use our …
Eyes
Hands
Bodies
Tools
Nearby objects
To Think
Prove:
All three medians of a triangle
always intersect at a single point.
Can you do it just in your head?
Do the medians intersect at a single point?
Conceive then Imagine
Do the medians intersect at a single point?
Triangle
Medians Intersect
Imagination is fallible
All three medians intersect at a point.
Easier if you create an external representation
Conceive then
Externalize
Create 3rd median
Still too hard to project reliably so create
THINKING IS OFTEN INTERACTIVE
Claim One
PROJECT CREATE INTERACTIVE
STRATEGY
TWO: PROJECTION CAN BE IN
ANY MODALITY
Claim
Look and use your hands to feel its width
Kinesthetic projection
Look and use your hands to feel its width
Kinesthetic projection
Anchored on visually perceived object
She sells seashells by the seashore
Read this silently
Read this silently
She sells seashells by the seashore
Auditory projection onto words
WE PROJECT IN MANY
MODALITIES
Claim Two
KINESTHETIC, SOUND, VISION . . .
THREE: WE CREATE STRUCTURE
TO
PROJECT ONTO
IN MANY MODALITIES
Claim
Thinking through Gesture Mental Abacus
Source: Brooks et al, Gesture in Mental Abacus Calculation, 2012
Visual projection onto hand movement or
Kinesthetic projection onto hands
Numbers are encoded visually rather than as sounds or words
Thinking through Body Movement
Work done with Paul Maglio 1994
Where should this
piece go?
Thinking by
moving objects
Work done with Paul Maglio 1994
Physical Rotation
is faster than
Mental Rotation
Work done with Paul Maglio 1994
Physical Rotation
reveals different
shapes
We project the location
Work done with Paul Maglio 1994
Will these
Make this
Solving it in the world is more cost-effective
• More reliable
• Usually faster
• Less effortful
• Scales up better
Oops
WE CREATE STRUCTURE TO PROJECT
ONTO
IN MANY MODALITIES
Claim Three
HAND MOTION, SOUND, MOVING OBJECTS . . .
Interim Summary
We think interactively through
projection creation
Thinking often involves building models or
simulations
– to make sense of things or
– to think things through
Interim Summay
We think interactively through
projection creation
Thinking often involves building models or
simulations
– to make sense of things or
– to think things through
Interim Summary
This interactive strategy is multimodal
Gestures + Mental Imagery
(projection)
Body Movement + Mental Imagery “
Manipulate objects + Mental Imagery “
Writing + Mental Imagery
Drawing + Mental Imagery “
FOUR: INTERACTIVE THINKING REQUIRES A TIGHT
TEMPORAL COUPLING BETWEEN INTERNAL
AND EXTERNAL PROCESSES
Claim
Hand - ‘Mind’ Coordination
Tight Temporal Coupling
Only useful for moments
Kinesthetic imagery - projection
Thinking involves tight temporal coupling between components
Executes this
Thinking this helps shape performance
Dancer: Noah Gelber
Tight’ish temporal coupling
• ‘Read these words’
– inner speech is tightly coupled
• Write what you’re thinking
– to build on your writing interactively inner processes cannot
decouple from outer ones
• write read write
• Conversation – joint activity breaks down if delays are
extended
TIGHT TEMPORAL COUPLING BETWEEN
INTERNAL AND EXTERNAL
Claim Four
MENTAL IMAGERY MOVEMENT
FIVE: WHEN WE EXTERNALLY MODEL WE
MENTALLY PROJECT TO SOME ASPECT
OR
DIMENSION OF THE TARGET THING
Claim
When making a caricature you think of some aspect of the real thing
Models are simpler than the thing modeled
WE MENTALLY PROJECT TO SOME ASPECT
Claim Five
LOOK AT IMAGE SEE JOWLS, DETERMINATION
Agenda
✔ Main Theoretical Claims
Thinking through the Body
Study of Dance Making
Thinking through Instruments
Action vs. Observation
Thinking through Ordinary Objects, Bottles, Representations, …
Agenda
✔ Main Theoretical Claims
Thinking through the Body
Study of Dance Making
Thinking through Instruments
Action vs. Observation
Thinking through Ordinary Objects, Bottles, Representations, …
Case Study
Thinking with our body
Body Mediated Cognition
How we studied
dance making
Timeline – ‘Dyad’ 2009
Wayne McGregor
Jan 26th Feb 13th
UCSD
Aug 25th Sept 7th
London
Oct 13th
Premiere Total = 6 weeks
Random Dance
Timeline – ‘Far’ 2010
Sept 13th Nov 13th
London
Nov 17th
Premiere Total = 6 weeks
Timeline – ‘Undance’ 2011
Aug 30th Oct 23rd
London
Nov 30th
Premiere
Total = 6 weeks
Timeline – ‘Atomos’ 2013
July 21st Oct 8th
London
Oct 9th
Premiere
Total = 11 weeks
Observation Set-Up - UCSD
Rear2
Rear1
W Cam
Daily2
Daily1
Mandeville
Observers
Mandeville Auditorium Cal-IT2
Daily 2 W Cam (b)
Daily 1 (a)Rear 1 (b)
Rear 1 (c)Rear 2
Cal-IT2
W Cam (a)
ArtistEntrance
Rear 1 (a)
Daily 1 (b)Rear 1 (a)
Student'sEntrance
Dyad UCSD
Field Notes Observations 5 hours/day 6 -11 weeks
Time, Activity code, Description of activity
Interviews Before and after session
2 hours per day, for 23 days
Interviews
After session each day usually in two’s
Data Collected at Each Capture
• 20 TB of video of dancers and choreographer
• Dozens of interviews with choreographer and
dancers
• Still images
• Dancer notes
• Associate choreographer’s notes
• Student notes of ongoing activity
• Music used
MARKING in dance
Thinking with the body
Marking - during practice
What is Marking?
A dance phrase is practiced, explored or reviewed in a less
energetic manner than doing it ‘full-out’.
Small marking
Marking for time
Marking
• Imperfect model of real phrase
• a sketch, abstraction
• Attend to specific aspect of a movement
• Dancers often practice real phrases by marking
Marking: a universal phenomenon
• Tennis swing – work on aspect
• Cello – on the arm
• Staged Plays – an Italian run-through
• Imperfect modeling – aspectual – as a
learning/practice technique
Teachers have students swing without hitting a ball in order to grasp shapes or make small adjustments
Not even a tennis racket
Planning grips and placements in rock
climbing by marking when on the ground
Similar Phenomena
Planning a downhill run when skiing moguls
Similar Phenomena
Antoine: Full
Antoine: Large marking
Antoine: Small marking
Agnes: Full vs. Marked
I’m not using energy but I’m thinking what’s my pathway
Agnes: Large vs. Small marking
Tombe Pas de Bouree
Experiment to explore value of marking
Which condition facilitate learning most?
Mark Full-Out Simulate in the head
Hypothesis
Mark
Full-Out
Simulate in the head
>
>
Experimental Design
MEASURE MEASURE PRACTICE TEACH
Experimental Design
Mark
Lie on floor
Full-Out
Trial 2
Trial 3
Trial 1
Mark
Lie on floor
Full-Out Mark
Lie on floor
Full-Out
Group 1 Group 2 Group 3
Performance Measures
1 2 3 4 Technicality Memory Dynamics
Precision of positions Completeness of detail Speed, Force, Acceleration
Timing
Tempo, duration
RESULTS
Higher is better
P = .018
Marking was best!
Measure Mark >Full Full>Mark Mark>Sim Full>Sim
Memory .7334 < .0001 < .0001
Technicality .0029 < .0001 .0005
Timing .0194 < .0001 < .0001
Dynamics - .145 .0003 < .0001
Mem, Tech, Timing .0189 - < .0001 < .0001
Higher is better
P Values
Results by Measured Dimension
Upshot: Marking Study
• Marking is a movement reduction
system
• Other movement reduction systems that
might facilitate thought:
• Whispering or Subvocalizing
• Gesturing
• Physical miming with objects
How Does It Work?
Marked aspect anchors
projection
Marked phrase
‘Imagery”
In world
In mind
Movement
Projection in mind
PROJECTION
Projection vs. Imagination
Reality oriented
See what is present
Perception
Augment reality
Anchored Imagery
Virtual Reality
Imagery has no size or location
Projection Imagination
Projection: Claims
1. Mental Projection is more powerful than
mental imagery alone
2. We can project beyond what we can
readily imagine
3. External structure helps us
Experiment to explore projection
Tic Tac Toe
Blank Sheet
Imagination Condition
Projection Condition
Projection Condition - 2
X O
Tic Tac Toe experiment – 3 by 3
Projection conditions
Blank Sheet
Imagination Condition
Blank Table Table + X O
Experimental Conditions
X O
Training
1 2 3
4 5 6
7 8 9
Blank Sheet
Within Subject Design, with Practice Phase
Imagination Condition Practice
Blank
X O
Projection conditions
Table Table + X O
Projection ≠ Memory Offload of State
Board remains the same over time
Memory not offloaded because no persistent structure
X
O
4,8 4,9
4,6
4,0
4,5
5,0
5,5
6,0
Blank Table Table + XO
Overall Means
Not significant N = 25
Secs
X O
Lower is better
Results 3 by 3
4.9s ≈ 4.8s
Surprise!
≈
Table is no faster than blank
Results 3 by 3: Individual Differences
4,4
5,8
4,7
4,0
4,5
5,0
5,5
6,0
Blank Table Table + XO
Significant p = .002
X O More than half were significantly better using
Imagination alone
Secs
X
Anchoring Costs
Anchoring = Spatializing a component of mental image
X Placement costs
Do benefits ever overcome
anchoring costs?
If the imagery task is hard
enough everyone will benefit
Conjecture:
4 by 4 Board
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
4 by 4 Experiment: harder imagery task
Imagination Projection
Practice
Blank Table Table + X O
3 Conditions
Blank Page
X O
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
Results 4 by 4
Table is worth the cost of coordination
If task is hard enough
Table faster than XO p=.01
N = 25
4 by 4 Mean Time per move
p = .002
> p = .002
9.1s 10.7s
Is it better for everyone?
Bad visualizers are helped much more!
Strong grid > blank p = .11
Weak grid > blank p = .01
*
Lower is better * Weak 10.4 < 14.1 Strong 9.3 < 10.1
>
Strong trending to benefit
Blank XO Grid Blank XO Grid Blank XO Grid
Upshot: Projection
• Projection: real process
– distinct from perception and imagination
• As problems get harder we cannot easily
imagine the answer
• So we rely on creating external structures
to project onto and scaffold imagination
SIX: IF WE CAN THINK WITH OUR BODIES
THEN WE CAN THINK WITH THINGS
THAT WE ARE HIGHLY PRACTICED
WITH
Claim
Agenda
1. Thinking with the Body
Study of Dance Making
Thinking with Tools
Action vs. Observation
Thinking with Ordinary Objects, Representations …
Conclusions
Thesis
Eyes
Hands
Bodies
Tools
Nearby objects
To help us think
We Use our
Players can Mark with their Instrument
As if it is their body
So ... Experts can think with
their
instruments
Can they have the same
thoughts without?
Why an expert might have the
same thought without playing
Mirror Neurons!
Super Experts
• Play enough, you eventually simulate playing without an
instrument in hand - i.e. internalization
– Instrument-mediated cognition
is possible without the instrument
• Perlman’s auditory perception
is so expert when he listens
it is as if he is physically playing
Neural support that observing is
enough
• Motor resonance, mirror neuron, action observation network
– Watching or listening stimulates covert actions as if dancing or playing
• Covert actions involve motor planning, just as overt action does
• The real difference is that covert action does not activate muscular control
• So experts simulate playing
Support from Enactive Perception
• Enactive account: observer sees the counterfactual futures in the present – what it would look like if I moved, turned it, played it …
• Observer’s phenomenological experience includes possible ways a phrase may be continued
So in auditory perception agents
enact a possible future
YET
Do mirror neurons simulate at a fine
enough resolution?
empirical question … but
Listening is not Performing
physicality of the piece
Performer Situation ≠ Observer
Situation • Performer has responsibility to succeed
• observer doesn’t
• Performer must decide: how to attack a note, its mood,
emotionality
• These concepts are ad hoc, situated, and embodied
• unavailable even to an an expert observer.
• Hence agents project a future that is conceptually and
experientially richer than the future projected by an
observer
• Simulation is a lower resolution than actual playing
With a violin in hand musical engagement is
mediated differently than in imagination and
active listening
Instrument mediated conclusion
Agenda
1. Thinking with the Body
Study of Dance Making
Thinking with Instruments
Action vs. Observation
Thinking with Ordinary Objects, Representations …
Conclusions
Thesis
Eyes
Hands
Bodies
Tools
Nearby objects
To help us think
We Use our
Thinking with Illustrations by deforming
them on a stretchy surface
D’Arcy Thompson’s famous fish geometry
Distorting the plane equals distorting growth rates – explains different forms
Stretching, twisting, shearing
Remember this …?
Solving it in the world is more cost-effective
• More reliable
• Usually faster
• Less effortful
• Scales up better
Cognition flows to wherever costs
are lower
Internal Processes
External Processes
In closely coupled system, process and structure migrates
to wherever computational and physical costs are lowest
External Processes
Physical is cheaper
Here processes flow to the physical
Thinking by interacting with 3D models
We use the model as tool for visualization – focusing attention on things that suddenly make sense
Driving thought forward
Objects in motion can do work for us
We can think causally with objects
If we interact we can:
• Compare rotation speeds
• Try out speeds
• Get a feel for force
• Visualize effects
Agenda
1. Thinking with the Body
Study of Dance Making
Thinking with Instruments
Action vs. Observation
Thinking with Ordinary Objects, Representations …
Conclusions
THINKING IS MULTI-MODAL
Bodies & Things
Eyes & Hands
Bodies & Kinesthetics
HIGHLY PERCEPTUAL Visual Probing
Kinesthetic Probing
USUALLY INVOLVES PROJECTION
INSIDE AND OUTSIDE
PROCESSES ARE TIGHTLY
COUPLED
OFTEN INVOLVES PHYSICAL OR
MENTAL SIMULATION OR
CONSTRUCTION
SIMULATION AND PROJECTION ARE
HARD SO INTERACTING IS USUALLY
PART OF THINKING
Eye hand mediated cognition
Body mediated cognition
Tool mediated cognition
External Rep’n mediated Cognition
Object mediated Cognition
WITH THE RIGHT STRUCTURES AND
MOVEMENTS WE CAN PUSH
THOUGHT FORWARD FASTER
158 slides
Thank you for concentrating!
Acknowledgements
Thanks to
Dafne Muntanyola
Wayne McGregor
Odette Hughes
Random Dance Company
Funding:
NSF grant: Distributed Creative Cognition in Dance
UCSD core grant
R-Research, London
UCSD classes
The End
Enough