sidewalk extraction using aerial and street view images
TRANSCRIPT
![Page 1: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/1.jpg)
Sidewalk Extraction Using Aerial and Street View Images
Xinyue Ye, Ph.D., Associate Professor
Department of Landscape Architecture and Urban Planning
& Department of Geography
Texas A&M University, Email: [email protected]
Editor-in-Chief, Computational Urban Science
Urban Data Science Lab, TAMU
![Page 2: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/2.jpg)
Acknowledgement
• Ning, H., Ye, X*., Chen, Z., Liu, T., & Cao, T. (2021) Sidewalk Extraction Using Aerial and Street View Images. Environment and Planning B. (in press)
• Ye, X. and Lin, H. (eds.) (2020) Spatial Synthesis: Computational Social Science and Humanities, Springer.
• Jamonnak, S., Zhao, Y., AL-Dohuki, S., Curtis, A., Ye, X., Kamw, F., & Yang J. (2020) GeoVisuals: a visual analytics approach to leverage the potential of spatial videos and associated geonarratives. International Journal of Geographical Information Science. doi: 10.1080/13658816.2020.1737700
• Ye, X. and Liu, X. (eds.) (2019) Cities as Social and Spatial Networks, Springer.
Urban Data Science Lab, TAMU
![Page 3: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/3.jpg)
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
![Page 4: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/4.jpg)
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
![Page 5: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/5.jpg)
Walking
Urban Data Science Lab, TAMU
![Page 6: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/6.jpg)
Which information is missed?
Urban Data Science Lab, TAMU
![Page 7: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/7.jpg)
Big Image Data
Urban Data Science Lab, TAMU
http://shadeparadenashville.blogspot.com/2013/12/sidewalk-project-3-walking-from.html
![Page 8: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/8.jpg)
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
![Page 9: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/9.jpg)
Semantic segmentation based on deep learning
• FCN (Long et al., 2015)
• SegNet (Badrinarayanan et al., 2017)
• U-Net (Ronneberger et al., 2015)
• PSP-Net (Zhao et al., 2016)
• LaU model (He et al., 2019)
• OCR model (Yuan et al., 2019)
Urban Data Science Lab, TAMU
![Page 10: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/10.jpg)
Road extraction from aerial image
• ResUnet on the Massachusetts roads dataset (Mnih and Hinton, 2010)
• SegNet to segment roads in the Massachusetts dataset and the Thailand Earth Observation System (THEOS) dataset (Panboonyuen et al. 2017)
Urban Data Science Lab, TAMU
![Page 11: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/11.jpg)
Sidewalk extraction from street view image
• Senlet and Elgammal (2012): a pixel-wise and a path-wise method to predict and connect the occluded sidewalks covering a university campus.
• Smith et al. (2013): random forest to identify sidewalk after extracting local and global features.
• Mattyus et al. (2016): fully connected deep structured networks and Markov random field.
• Wang et al. (2019):convert street view image into top-view representation.
Urban Data Science Lab, TAMU
![Page 12: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/12.jpg)
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
![Page 13: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/13.jpg)
sidewalk datasets
• field survey
• deriving from existing land cover data
• extracting from images
Urban Data Science Lab, TAMU
![Page 14: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/14.jpg)
workflow
Urban Data Science Lab, TAMU
![Page 15: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/15.jpg)
Study area (Delaware Valley Regional in Pennsylvania State)
Urban Data Science Lab, TAMU
![Page 16: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/16.jpg)
Training set and test set
Urban Data Science Lab, TAMU
https://medium.com/@abhigoku10/cv2019-papersummary-yolact-real-time-instance-segmentation-e62fa721957f
![Page 17: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/17.jpg)
A sample of the sidewalk training set. The DVRPC sidewalks dataset uses a short line to represent the
conjunction point with crosswalks, which is preserved in the training dataset.
Urban Data Science Lab, TAMU
![Page 18: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/18.jpg)
Missing sidewalks in aerial images between dangles
Urban Data Science Lab, TAMU
![Page 19: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/19.jpg)
Segmentation of street view images by pre-train PSPNet
Urban Data Science Lab, TAMU
![Page 20: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/20.jpg)
A panorama and its depthmap in Equirectangularprojection from Google Street View
Urban Data Science Lab, TAMU
![Page 21: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/21.jpg)
Reproject the segmentation to landcover map
Urban Data Science Lab, TAMU
![Page 22: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/22.jpg)
Extracting sidewalk centerlines from street view images. The centerlines were extracted merely from
landcover map for visual representation
Urban Data Science Lab, TAMU
![Page 23: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/23.jpg)
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
![Page 24: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/24.jpg)
Sidewalk detection results of YOLACT
Urban Data Science Lab, TAMU
![Page 25: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/25.jpg)
Precision and recall of segmentation results
Urban Data Science Lab, TAMU
![Page 26: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/26.jpg)
An occluded segment of sidewalks was restored using street view images
Urban Data Science Lab, TAMU
![Page 27: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/27.jpg)
A missing sidewalk segment of 85 meters was restored using street view images
Urban Data Science Lab, TAMU
![Page 28: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/28.jpg)
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
![Page 29: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/29.jpg)
https://www.nwnewsnetwork.org/post/washington-lawmakers-
ponder-rules-sidewalk-driving-delivery-robots
https://www.scientificamerican.com/article/out-of-the-way-human-delivery-robots-want-a-share-of-your-sidewalk
![Page 30: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/30.jpg)
Reality?
Urban Data Science Lab, TAMU
https://www.latimes.com/local/lanow/la-me-ln-fix-broken-sidewalks-20140818-story.html
![Page 31: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/31.jpg)
Urban Data Science Lab, TAMU
https://www.dailynews.com/2018/02/07/la-city-leaders-say-streets-sidewalks-are-broken-and-may-not-be-ready-in-time-
for-olympics/
![Page 32: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/32.jpg)
Urban Data Science Lab, TAMU
![Page 33: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/33.jpg)
Data Challenging Communities
33
![Page 34: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/34.jpg)
Urban Memory
34
query, navigation, and comparison of video, photo, audio, and text
over regions and locations
![Page 35: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/35.jpg)
Urban Memory
35
![Page 36: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/36.jpg)
36
Procedure
![Page 37: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/37.jpg)
37Procedure
![Page 38: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/38.jpg)
GeoVideo
● Location-based video capturer
● Without Internet connection
● Official website
○ http://vis.cs.kent.edu/GeoVisuals/mobile.html
● Appstore○ https://apps.apple.com/us/app/geovideo/id1492419929
● Google Play○ https://play.google.com/store/apps/details?id=com.nlapps.geovideo&hl=en
Apps
![Page 39: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/39.jpg)
39
Human in Cities
![Page 40: Sidewalk Extraction Using Aerial and Street View Images](https://reader030.vdocuments.us/reader030/viewer/2022012804/61bd25a861276e740b0fd73b/html5/thumbnails/40.jpg)
40
MethodologicalConceptual
Transformative Outcome
Human-Centered Built Envr. Science
Urban Computing
• Energy Research
• Hazard Reduction & Recovery
• Housing & Land Use
• Smart & Connected Cities/Health
• Spatial Decision Support System
• Transportation & Travel Behavior
Workflow Automation
Urban Systems and Communities in the 21st Century
Text as Data
Emerging AI, ICT, and location-aware technologies
Visual Analytics
Data Fusion
Human Dynamics
Image and Video Analytics
Real-time SystemSpatial Econometrics
Space-time Analysis
Collaborative Research Plan
Turn space to place
from concept to deliverables