soccer video analysis
DESCRIPTION
Soccer Video Analysis. EE 368: Spring 2012 Kevin Cheng. Goal. To detect and track key features needed to interpret events in a Soccer game from a video. Clip. Overview. Frame Pre-Processing. Input Image. Field Detector. Player Detector. Ball Detector. Field Detector. - PowerPoint PPT PresentationTRANSCRIPT
Soccer Video AnalysisEE 368: Spring 2012
Kevin Cheng
To detect and track key features needed to interpret events in a Soccer game from a video
Goal
Clip
Player DetectorField Detector Ball Detector
OverviewFrame Pre-Processing
Input Image
Field Detector
Harris Corner Detector
Input Image
RGB Map Assignment
Input Image
Convex Hull of Largest Region
Hough Transform
Hough Mask
Field Color Map Creator
Field Mask
Field detector is “trained” on a set of images with known RGB values, using a 16x16x16 color map
Field Detector
Harris Corner Detector
Input Image
RGB Map Assignment
Input Image
Convex Hull of Largest Region
Hough Transform
Hough Mask
Field Color Map Creator
Field Mask
For each frame, using the field color map, each pixel is labeled as “field” or “non-field”
Field Detector
Harris Corner Detector
Input Image
RGB Map Assignment
Input Image
Convex Hull of Largest Region
Hough Transform
Hough Mask
Field Color Map Creator
Field Mask
The largest region is considered where the field lies. A mask is created from the convex hull of the region.
Field Detector
Harris Corner Detector
Input Image
RGB Map Assignment
Input Image
Convex Hull of Largest Region
Hough Transform
Hough Mask
Field Color Map Creator
Field Mask
Using a canny edge detector and hough transform, we attempt to extract the field lines to help extract information of the field location
Harris Corner Detector
Input Image
SIFT Keypoint Detector
Input Image
Keypoint Team Assignment
Redundancy Removal
Player Tracking
Player Positions
𝑧− 1
Team Color Decider
Player Detector• Using the first frame as a
keyframe, we determine the mean RGB values of the 2 teams.
• This is done by taking a histogram of the field, and finding the 2 highest bins that are NOT labeled as ‘field’ by the field map.
Player Detector• To find potential player
positions, we use a SIFT keypoint detector to narrow down the search regions
Harris Corner Detector
Input Image
SIFT Keypoint Detector
Input Image
Keypoint Team Assignment
Redundancy Removal
Player Tracking
Player Positions
𝑧− 1
Team Color Decider
Harris Corner Detector
Input Image
SIFT Keypoint Detector
Input Image
Keypoint Team Assignment
Redundancy Removal
Player Tracking
Player Positions
𝑧− 1
Team Color Decider
Player Detector
• Based on the RGB values around the SIFT keypoints, each keypoint is assigned to a team based on distance in RGB space.
• Keypoints whose calculated distance is too large are thrown out.
Harris Corner Detector
Input Image
SIFT Keypoint Detector
Input Image
Keypoint Team Assignment
Redundancy Removal
Player Tracking
Player Positions
𝑧− 1
Team Color Decider
Player Detector• Since a single player can
produce multiple SIFT keypoints, we implement a minimum distance connectivity map to remove redundant keypoints.
Keypoint# 1 2 3 4 5 6 7
1 0 133.0902 94.64143 18.02776 61.91123 61.91123 164.7938
2 133.0902 0 38.47077 115.1173 167.9047 167.9047 137.186
3 94.64143 38.47077 0 76.65507 132.8157 132.8157 131.5295
4 18.02776 115.1173 76.65507 0 72.13876 72.13876 153.675
5 61.91123 167.9047 132.8157 72.13876 0 0 225.6103
6 61.91123 167.9047 132.8157 72.13876 0 0 225.6103
7 164.7938 137.186 131.5295 153.675 225.6103 225.6103 0
Connectivity Map
• A connectivity map is created were the (i,j) entry is the pixel distance between keypoint i, and keypoint j.
• This matrix is square and symmetric
Keypoint# 1 2 3 4 5 6 7
1 0 133.0902 94.64143 18.02776 61.91123 61.91123 164.7938
2 133.0902 0 38.47077 115.1173 167.9047 167.9047 137.186
3 94.64143 38.47077 0 76.65507 132.8157 132.8157 131.5295
4 18.02776 115.1173 76.65507 0 72.13876 72.13876 153.675
5 61.91123 167.9047 132.8157 72.13876 0 0 225.6103
6 61.91123 167.9047 132.8157 72.13876 0 0 225.6103
7 164.7938 137.186 131.5295 153.675 225.6103 225.6103 0
Total 0 1 1 0 1 1 0
• All entries larger than 40 are tagged as duplicates.• Here we see that keypoints (2,3) should be
merged, and that (5,6) are duplicates.
Connectivity Map
Keypoint# 1 2 4 5 6 7
1 0 133.0902 18.02776 61.91123 61.91123 164.7938
2 133.0902 0 115.1173 167.9047 167.9047 137.186
4 18.02776 115.1173 0 72.13876 72.13876 153.675
5 61.91123 167.9047 72.13876 0 0 225.6103
6 61.91123 167.9047 72.13876 0 0 225.6103
7 164.7938 137.186 153.675 225.6103 225.6103 0
Connectivity Map
• Systematically, we remove duplicates starting from the highest connected keypoint.
Keypoint# 1 2 4 5 7
1 0 133.0902 18.02776 61.91123 164.7938
2 133.0902 0 115.1173 167.9047 137.186
4 18.02776 115.1173 0 72.13876 153.675
5 61.91123 167.9047 72.13876 0 225.6103
7 164.7938 137.186 153.675 225.6103 0
Connectivity Map
• Systematically, we remove duplicates starting from the highest connected keypoint.
Keypoint# 1 2 4 5 7
1 0 133.0902 18.02776 61.91123 164.7938
2 133.0902 0 115.1173 167.9047 137.186
4 18.02776 115.1173 0 72.13876 153.675
5 61.91123 167.9047 72.13876 0 225.6103
7 164.7938 137.186 153.675 225.6103 0
Connectivity Map
• The remaining keypoints are passed along as the player positions detected in the current frame
Harris Corner Detector
Input Image
SIFT Keypoint Detector
Input Image
Keypoint Team Assignment
Redundancy Removal
Player Tracking
Player Positions
𝑧− 1
Team Color Decider
Player Tracking• Since it cannot be assumed
that we detect all players in every single frame, we implement a tracking algorithm.
• This algorithm once again utilizes a connectivity map to match current detections with history element.
1 2 3 4 5 6 7 8 9 10 11
1 0 542.04 326.60 224.26 435.08 127.94 52.30 356.31 161.00 265.56 437.06
2 53.17 500.1 285.17 178.68 389.41 103.71 1 310.21 117.68 229.45 395.82
3 224.26 322.32 109.90 0 211.01 119.95 179.68 132.06 63.68 77.96 219.15
4 357.3 195.84 57.13 133.06 78.25 245.94 312.21 1 196.45 122.60 101.85
5 435.08 123.17 117.39 211.01 0 320.86 390.41 79.25 274.08 188.44 60.34
6 543.07 1.05 216.48 323.37 124.19 421.1 502.13 197.82 383.61 281.00 106.16
7 129.12 419.11 205.57 119.40 319.99 1.20 105.59 244.17 64.14 139.43 314.02
8 163.00 380.60 165.52 61.70 272.09 65.60 120.61 193.46 2 113.38 276.19
9 437.06 105.15 110.70 219.15 60.34 315.03 396.80 102.60 278.16 175.13 0
10 323.58 218.47 3.01 107.01 120.09 203.55 283.14 58.00 164.48 68.41 113.71
History
Now
Player Tracking
• The (i,j) entry compares the pixel distance between the ith current detection and jth history element
• Moving from left to right, history elements with their closest current keypoints in a one-to-one matching
Harris Corner Detector
Input Image
SIFT Keypoint Detector
Input Image
Keypoint Team Assignment
Redundancy Removal
Player Tracking
Player Positions
𝑧− 1
Team Color Decider
Player Tracking
• For each player, we assign a counter that keeps track of its detection history:– If the history element is matched
with a current element, we increment the counter by 2 (max of 15).
– If a history element does NOT find a match in the current frame, we decrement the counter by 1 (minimum of 0).
– Only players whose counter is greater than 5 are displayed.
– If a player’s counter drops below 0, it is removed from the history
Harris Corner Detector
Input Image
Harris Corner Detector
Input Image
Hough and White Mask
Gaussian Weighting
Max Value
Ball Position
𝑧− 1
Ball Detector• The ball detector is based
on corner detection since a small round object should have high “cornerness”.
Harris Corner Detector
Input Image
Harris Corner Detector
Input Image
Hough and White Mask
Gaussian Weighting
Max Value
Ball Position
𝑧− 1
Ball Detector• To filter out all other corners in the
image, we only look at corners with white RGB values.
• Another source of white corners in the image are the white field lines. So we also mask out corners that lie on the detected hough transform.
Harris Corner Detector
Input Image
Harris Corner Detector
Input Image
Hough and White Mask
Gaussian Weighting
Max Value
Ball Position
𝑧− 1
Ball Tracking• Lastly, for ball tracking, we
weight the corner metric values by a gaussian centered at the last detected spot of the ball.
• After the gaussian weighting, we take the maximum remaining value to be the new location of the ball.
• Strengths:– Has a > 90% detection and tracking rate of – Built in robustness to false-positives and
false-negatives• Weaknesses:– A little bit jittery from frame to frame due to
keypoint based detection.– Will falsely assign referees onto a team.
Results: Player Detector
Results: Player Detector
• Strengths:– Maintains track of ball very well
• Weaknesses:– Requires to see the ball in every frame– Partial ball obscuring could cause tracker to
latch onto another local maximum.
Results: Ball Detector
Results: Ball Detector
Timing Analysis• 2.63 seconds per frame.
• Field Detector: 46.6% (1.15 s/f)– .27 s spent in applyMap– Likely due to poor MATLAB
programming.– Canny edge and Hough transform
also big contributors
• Player Detector: 15% (.40 s/f) – SIFT detector: .28 s/f
• Ball detector: 10.9% (.27 s/f)
• Field Detection:– Add ability to detect where in the field you are
using field lines and features.• Player Detection and Tracking:– SIFT is a large computational bottleneck. Are there
more viable alternatives? • Ball Detection and Tracking:– Improve handling of cases where ball is obscured
for a short amount of time.• Analysis and interpretation of detected events.
Further Work
