the recognition of human movement using temporal templates
DESCRIPTION
The Recognition of Human Movement Using Temporal Templates. Liat Koren. Lecture subjects. Introduction Prior work The Temporal Templates Usage example. Introduction. Computer vision trends Less image or camera motion More on labeling of action Reasons More computational power - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/1.jpg)
The Recognition of Human Movement
Using Temporal Templates
Liat Koren
![Page 2: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/2.jpg)
- 2 -
Lecture subjects
• Introduction• Prior work• The Temporal
Templates• Usage example
![Page 3: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/3.jpg)
- 3 -
Introduction
• Computer vision trends
– Less image or camera motion
– More on labeling of action
• Reasons
– More computational power
– Wireless application
– Interactive environments
![Page 4: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/4.jpg)
- 4 -
Introduction – cont.
• Recent efforts are in Three Dimensional object reconstruction– Assuming it will have to be used in the recognition
of human motion.
• This article claims otherwise– View-based approach– Direct recognition
![Page 5: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/5.jpg)
- 5 -
Motivating Example
![Page 6: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/6.jpg)
- 6 -
Motivating Example
![Page 7: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/7.jpg)
- 7 -
Motivating Example
• Static pictures– Hard to recognize.
• Sequence on motion– Human can recognize without three dimensional
reconstruction.
• Conclusion– It is possible to recognize movement using only
the motion itself.
![Page 8: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/8.jpg)
- 8 -
• Process– Recover the pose of the person at each time
instant using a 3D model.– The model’s projected image should be as close
as possible to the object(e.g. edges of body in the image)
• Drawbacks– Complicated process– Human interference is usually required– Special imaging environment
![Page 9: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/9.jpg)
- 9 -
2D Based recognition
• Action is a sequence of static poses of
object.
• Requires
– Normalization
– Removal of background
![Page 10: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/10.jpg)
- 10 -
Wilson and Bobik’s approach• Actions are usually hand gestures• Representation
– Actual image– Grayscale– No background
• Benefits:– Hand appearance is fairly similar over a wide range of
people
• Problems– Actions that include the appearance of the whole body
are not visually consistent across different people.
![Page 11: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/11.jpg)
- 11 -
Yamato’s et al. approach• Representation
– No background– Black and white silhouettes
• Matching– Vector quantize – Usage of a mathematical method
• Benefits– Help handling the variability between people
• Problems– Disappearance of movement inside the silhouette
![Page 12: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/12.jpg)
- 12 -
Summery of prior work
• Action is a sequence of static poses.• Requires individual features or properties that
can be extracted and tracked from each frame.
• Recognition of movement from a sequence of images is a complicated task.
• Usually requires previous recognition and segmentation of the person.
![Page 13: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/13.jpg)
- 13 -
Motion based recognition• Attempt to characterize the motion itself
without reference to the underlying static poses of the body.
• Possible approaches– Blob like representation– Tracking of predefined regions (e.g., legs, head,
mouth) using motion.• Face expression patches• Whole body patches
– Measure typical patterns of muscle activation
![Page 14: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/14.jpg)
- 14 -
Terms
• Movement– where – motion has occurred in image
sequence. • MEI – Motion Energy Image
– how – the motion is moving.• MHI – Motion History Image
+
Temporal Templates
![Page 15: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/15.jpg)
- 15 -
Temporal Templates
• Representation of movement– View specific– Movement is motion in time– Vector image that can be matched against stored
representations of movements.
• Assumptions– Background is static– Camera movements can be removed– Motion of irrelevant objects can be eliminated
![Page 16: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/16.jpg)
- 16 -
Motion-Energy Imageswhere did the movement occurred ….
![Page 17: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/17.jpg)
- 17 -
Motion-Energy Images
• Notice that:– If τ is very big, all the differences are accumulated– Τ has a vast influence on the temporal
representation of a movement.
![Page 18: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/18.jpg)
- 18 -
Motion-Energy Images
• Smooth change in the viewing angle causes a smooth change in the viewed image, thus coarse sampling of the viewing circle is enough (30°)
![Page 19: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/19.jpg)
- 19 -
Motion-History Images
• Intensity of a pixel represents the temporal
history in that pixel.
• Newer movement is brighter.
![Page 20: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/20.jpg)
- 20 -
Motion-History Images
• A time-window of size τ is used – movement older than τ is ignored.
• The results of the article uses a simple
replacement and decay operator:
Notice that MEI can be calculated out of MHI by paintingin white any non-black pixel
One may wonder, why not use only
MHI ?Answers will be
given later…
![Page 21: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/21.jpg)
- 21 -
MEI and MHI in a nutshell
• MEI and MHI are two vector images designed to encode a variety of motion properties.
• Benefits in this representation is that the calculation is recursive, thus only up-to-date information need to be stored, making the computation both fast and space efficient.
![Page 22: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/22.jpg)
- 22 -
Matching Temporal Templates
• Collect training examples of each movement from a variety of viewing angles.
• Compute statistical representation of the MHI/MEI images (Hu moments)
• Given an input movement:– Calculate a statistical representation– Use mahalanobis distance to find a stored
movement, that is the nearest to the input.
![Page 23: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/23.jpg)
- 23 -
Mahalanobis Distance Example
![Page 24: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/24.jpg)
- 24 -
Reasoning for the algorithm
• Mahanobis distance provides:– Good matching as shown in the results of the article.
– Simple calculation which makes real-time applications feasible.
• Hu moments allow representation of images, that is invariant to scale or translation.One problem with Hu moments is that: “Hu moments are difficult to reason about intuitively” (the authors)
![Page 25: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/25.jpg)
- 25 -
Testing the system
18 exercises performed by experienced aerobicinstructor.
MEIs are on the bottomrows.
![Page 26: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/26.jpg)
- 26 -
Why both MHI and MEI ?
Because MHI and MEI
perceive two different
characteristics of the
movement (the “where”
and the “how”) they look
different ,and thus, both
essential.
![Page 27: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/27.jpg)
- 27 -
First experiment• Input 30° left of the subject• Match against all seven
views of all 18 moves• 12 out of 18 are correctly
recognized
0°
30° 60° 90° 120°150°
180°
![Page 28: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/28.jpg)
- 28 -
Analyze the results of 1st exp.
Move 13 in 30 ° Move 6 in 0 ° The correct match
input
false correct
![Page 29: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/29.jpg)
- 29 -
Combining multiple views
• Two cameras with orthogonal views• Minimize the sum of the mahalanobis
distance between the two input templates and two stored views of movement that have 90° between them.
• Hidden assumption: we know the angular relationship between the cameras.
![Page 30: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/30.jpg)
- 30 -
Second Experiment
0°
30° 60° 90°120°
150°
• Input with two cameras:• 30° left of the subject• 60° right of the subject
• Match against all seven views of all 18 moves
• 15 out of 18 are correctly recognized
![Page 31: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/31.jpg)
- 31 -
Analyze the results of 2nd exp.
Move 16 Move 15 The correct match
input
false correct
![Page 32: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/32.jpg)
- 32 -
Segmentation and Recognition• Problem : speed of performance is different
among different people.• Solution: Segmentation
– When training the system, calculate τmax and τmin for each movement.
– Use algorithm to match over a wide range of τ.
![Page 33: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/33.jpg)
- 33 -
Problems
• Problems with current system– One person partially occludes another
• Solution: Use several cameras
– More than one person appears in the view point
• Solution: use a tracking bounding box
![Page 34: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/34.jpg)
- 34 -
More Problems• Motion of part of the body is not specified
during a movement– Possible solutions
• Automatically mask away regions of this type of motion• Always include them
• Camera motion– Rather easy to eliminate since camera motion is
limited.
• Person is performing the movement while locomotion
![Page 35: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/35.jpg)
- 35 -
The KidsRoom: An Application• room is aware of the children (at most 4)• The room takes the children
to a story.• The room’s reaction is influenced
by the actions of the children.• Current story : adventurous tour to monster land• In the last scene the monsters teach the children to
dance.• Then, the monsters follow the children if they perform
movements they “know”• The narration coerces the children to room locations
where occlusions is not a problem
![Page 36: The Recognition of Human Movement Using Temporal Templates](https://reader035.vdocuments.us/reader035/viewer/2022062807/56814fe1550346895dbda878/html5/thumbnails/36.jpg)
- 36 -