© GEOMAR, Anne Jordt, Kevin Köser, 2015
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Computer Vision – Established Methods and
their Adaptation to Underwater Imaging
Anne Jordt
August 19th, 2015
Rostock, Germany
1 DeepSea Monitoring, GEOMAR Helmholtz
Centre for Ocean Research 2 Multimedia Information Processing Group, Kiel
University
1,2
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Helmholtz Centre for Ocean Research Kiel • Foundation by public law • Member of the Helmholtz Association of German Research
Centers • Budget: 60 Mio. Euro, about 30 mio. institutional funding, 30
mio. project funding • Staff: 750, about 400 scientists • Close relationship to Kiel University Kiel: joint professorships,
curricula, large research projects: cluster of excellence „The future ocean“, collaborative research projects 574 & 754
GEOMAR
Marine Research with tradition and innovation
Dr. Andreas Villwock
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Major research topics From the deep-sea to the atmosphere
Helmholtz Centre for Ocean Research Kiel
Dr. Andreas Villwock
• The role of the ocean in climate change: temperature and sea level rise, extreme events, CO2 budget
• Anthropogenic impact on marine ecosystems: food webs, ecosystem and climate change, ocean acidification, overfishing, aliens
• Marine resources: natural substances from the sea, gas hydrates, mineral resources
• Plate tectonics and natural hazards: subduction zones volcanism, and tsunamis
© GEOMAR, Anne Jordt, Kevin Köser, 2015
4
- Introduction
- Features and Feature Matching
- Geometry of Image Formation
- Calibration
- Structure from Motion
- Dense Stereo
- Conclusion
Outline
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Computer Vision – 3D Reconstruction
5
Bundler reconstructs 3D information from www.flickr.com large image sets (image from
http://www.cs.cornell.edu/~snavely/bundler/).
Introduction
Snavely et al. 2007
Example images bundler have been removed in this version
due to copy right reasons.
Please refer to the link below.
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Computer Vision – Motion Capture
6
Motion capturing for movies and computer games (image from
http://www.businessinsider.com/benedict-cumberbatch-motion-capture-smaug-hobbit-
2014-10?IRT)
Introduction
Example image for motion capture has been
removed in this version due to copy right reasons.
Please refer to the article linked below.
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Computer Vision – Autonomous Driving
7
Retrieving 3D information of the scene in front of a car for autonomous driving (image
from Badino et al. 2009).
Introduction
Badino et al. 2009
Example image of SGM and Daimler’s stixel world has been
removed in this version due to copy right reasons.
Please refer to the paper.
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Computer Vision – Autonomous Driving
8
Scene understanding through segmentation and tracking (image from http://www.6d-
vision.com/scene-labeling).
Introduction
Scharwächter et al. 2013
Example image of scene understanding has been removed in
this version due to copy right reasons.
Please refer to the webpage.
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Computer Vision – Mars Rover
9
Photo mosaicking on Mars (image from
http://www.nasa.gov/mission_pages/msl/images/index.html)
Introduction
Example images of the Mars Rover and one of the photo
mosaiks have been removed in this version due to copy right
reasons.
Please refer to the link below.
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Computer Vision – Face Detection
10
Automatic face detection in images (images from
http://cs.brown.edu/courses/cs143/2011/results/proj4/psastras/)
Introduction
Example images of face recognition have been removed in this
version due to copy right reasons.
Please refer to the link below.
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Effects of Water on Image Formation
11
• light traveling through
water is attenuated and
scattered -> effects on
image color
• light entering the
underwater housing is
refracted -> effects on
image geometry
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Scientific Applications - Geology
12
Left: underwater volcano, near Cape Verdes, water depth ca. 3500m (ROV team,
GEOMAR). Right: hydrothermal vent, Middle Atlantic Ridge (ROV team, GEOMAR).
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Scientific Applications - Archaeology
13
Left: Hedvig Sophia shipwreck (Florian Huber, Kiel University). Right: human skull
found in underwater cave system in Yucatan, Mexico (Christian Howe, Kiel University).
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Scientific Applications – Ocean and Atmosphere
14
Natural methane release (North Sea) and lab gas experiments
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Scientific Applications – Habitat Mapping and Monitoring
15
Left: cold water corals (Jago team, GEOMAR).
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Scientific Applications - Biology
16
Left: sloth skeleton in cave system in Yucatan, Mexico (Uli Kunz, Kiel University).
Right: shrimp (GEOMAR).
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Other Applications
17
- construction, e.g. harbor and bridge construction, oil rig maintenance
- deep sea mining, e.g. manganese nodules, massive sulfides
Left: Manganese nodules on seafloor (GEOMAR). Right: Manganese nodule
(GEOMAR).
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Capturing Underwater Images - Vehicles
18
• Remotely Operated Vehicle (ROV)
• tethered to the ship usually with real time video
data transmission
• equipped with thrusters (steered from pilot on
the ship)
• usually provides lighting
Top: ROV Kiel 6000 (GEOMAR, depth 6000 m)
Bottom: OpenROV, self-made, low cost ROV that
can be ordered by anyone and needs assembly
(water depth max. 50 m) from: www.openrov.org.
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Capturing Underwater Images - Vehicles
19
• Autonomous Underwater Vehicle (AUV)
• can navigate on pre-defined path autonomously
• equipped with thruster and different hardware for taking samples, not
necessarily cameras
• if equipped with cameras often strobe lights
• limited power supply
Image: AUV Abyss (GEOMAR). Equipped with sonar systems and digital stills
camera. Length: 4 m, depth rating 6000 m
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Capturing Underwater Images - Divers
20
• require extensive training
• equipped with cameras and lights
• limited diving time and depth
• humans have far better
capabilities of reacting to
environment
Image: scientific diver from Kiel
University with camera, light, and
scooter (Florian Huber).
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Capturing Underwater Images - Lighting
21
Lighting underwater scenes:
- sunlight in water depths close to surface
- in deep water lighting from ROVs and AUVs
- power consumption
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Capturing Underwater Images - Cameras
22
Different camera systems in pressure
housings:
- consumer hand-helds
- cameras used by divers
- cameras used on ROVs and AUVs
- GoPros
- OpenROV with cylinder housing
Cylindrical from:
www.openrov.org Dome port
Flat port
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Nuisances
23
Left: floating particles. Right: sunlight caustics caused by surface waves (images from
http://webee.technion.ac.il/people/yoav/research/flicker.html).
Swirski et al. 2007
Introduction
Example images of marine snow and flickering caustics have been
removed in this version due to copy right reasons.
Please refer to the link below.
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Challenges
24
• difficult imaging situations i.e. danger of water leakage, water pressure, salinity,
weather, difficult data transfer, bad visibility
• technical effort and complexity are MUCH higher than in air, especially compared to
lab environments
• images often captured on scientific cruises by people from other disciplines
• expensive; images cannot be taken again; often something missing (e.g. calibration)
• special problems with manual steps, but huge amounts of data, and growing
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
References
25
R. Szeliski, Computer Vision Algorithms and Applications, Springer 2011.
N. Snavely, S. M. Seitz, R. Szeliski. Modeling the World from Internet Photo
Collections. International Journal of Computer Vision, 2007.
H. Badino, U. Franke, D. Pfeiffer. The stixel world – a compact medium level
representation of the 3d-world. Pattern Recognition, 2009
T. Scharwächter, M. Enzweiler, S. Roth, and U. Franke. Efficient Multi-Cue Scene
Segmentation, In Proc. of the German Conference on Pattern Recognition
(GCPR), 2013
Y. Swirski, Y. Y. Schechner, B. Herzberg, and S. Negahdaripour, Stereo from
flickering caustics, Proc. IEEE ICCV (2009).
Introduction
© GEOMAR, Anne Jordt, Kevin Köser, 2015
Wrap up
26
• computer vision has the goal of automated image understanding
• multiple interesting applications for underwater vision
• different cameras, vehicles, and lighting situations to be taken into account
• two major effects on underwater image formation
• scattering and attenuation effects => appearance
• refraction at underwater housing => geometry
Introduction