1.1.Image-based Plant ModelingImage-based Plant Modeling
2.2.Image-based Image-based Tree Tree Modeling Modeling
Long Quan, Ping Tan, Gang Zeng, Lu Yuan, Jingdong Wang, Sing Bing Kang*
The Hong Kong University of Science and Technology* Microsoft Research
Image-based Plant ModelingImage-based Plant Modeling
Long Quan, Ping Tan, Gang Zeng, Lu Yuan, Jingdong Wang, Sing Bing Kang*
The Hong Kong University of Science and Technology* Microsoft Research
MotivationMotivation
• Plants are ubiquitous but difficult to model
– Complex geometry and topology
– Fine texture details
• Previous methods have limitations
– Manual intensive
– Unintuitive
– Lack of realism
FeaturesFeatures
• Only a handheld camera is used for capture
• Ability to capture complex geometry and texture
• User interaction is small
Overview of systemOverview of system…
…
3D 2D
Image Capture
Structurefrom Motion
Leaf Segmentation Leaf Reconstruction
Branch Editing
Plant Model
Render
Overview of systemOverview of system…
…
3D 2D
Image Capture
Structurefrom Motion
Leaf Segmentation Leaf Reconstruction
Branch Editing
Plant Model
Render
captured images(35-45 images)
cloud of reliable 3D points
Image Capture and Image Capture and Structure from MotionStructure from Motion
• Hand-held camera
• Use quasi-dense approach [Lhuillier & Quan 2005]
… …
Overview of systemOverview of system…
…
3D 2D
Image Capture
Structurefrom Motion
Leaf Segmentation Leaf Reconstruction
Branch Editing
Plant Model
Render
Leaf SegmentationLeaf Segmentation
• Goal: Segment 3D points and images into individual leaves
• Problem: Segmentation is subjective and ill-posed
• Our solution: Joint segmentation with user interaction
3D segmentation3D segmentation
• Automatic joint segmentation
– Graph model with joint 2D/3D distance
– Graph partition
• Interactive refinement
– User interface
– Graph update
graph model
3D segmentation3D segmentation —— —— Construct Construct 3D graph3D graph
Graph G = { V, E }:
V: 3D points recovered from SFM
E: each point connected to its K-nearest neighbors
3D segmentation3D segmentation —— —— Define joint 2D/3D distanceDefine joint 2D/3D distance
Distance between two nodes
– 3D distance : 3D Euclidean distance
– 2D distance
3 ( , )Dd p q
.p.q
( ) ( )( ) ( ) 3D 2D
3D 2D
d p,q d p,qd p,q = 1 - α + α
2σ 2σ
3 ( , )Dd p q
)(maxmax),(
],[2 ugqpd i
qpuiD
ii
p q
d2d(p,q)
= gradient of i-th image ig
3D segmentation3D segmentation—— —— GraphGraph ppartitionartition
By normalized cut [Shi & Malik 2000]
after 3D graph partition initial 3D Graph
2D segmentation2D segmentation
By two-label graph-cut algorithm
– FG: region covered by projected 3D points in a group
– BG: projections of all other points not in the group
……
……
Segmented 2D leaves Clustered 3D points
Interactive Interactive rrefinementefinement
• Click to confirm segmentation
• Draw to split and refine
• Click to merge
3D 3D ggraph raph uupdatepdate
By two-label graph-cut problem
– Min-cut algorithm
– Real-time visual feedback
before update split stroke after update
Overview of systemOverview of system…
…
3D 2D
Image Capture
Structurefrom Motion
Leaf Segmentation Leaf Reconstruction
Branch Editing
Plant Model
Render
Model-based leaf reconstructionModel-based leaf reconstruction
• Generic leaf extraction
• Leaf reconstruction
– Flat leaf fitting
– Boundary warping
– Texture extraction
– Shape deformation
Flat leaf fittingFlat leaf fitting
Estimate position, orientation, and scale by SVD decomposition of each 3D point set
Boundary warping & texturingBoundary warping & texturing
• Match leaf boundary to 2D segmentation boundary using iterative closest point (ICP) algorithm
• Crop texture after matching
leaf boundary
segmentation boundary
Shape deformationShape deformation
Move each vertex to the closest 3D point along normal of flat leaf
Overview of systemOverview of system…
…
3D 2D
Image Capture
Structurefrom Motion
Leaf Segmentation Leaf Reconstruction
Branch Editing
Plant Model
Render
Interactive Branch EditingInteractive Branch Editing
• Automatic reconstruction is difficult due to significant occlusion
• We rely on user to:
– Add branch
– Move branch
– Edit branch thickness (through radius)
– Specify leaf
Reconstruction statisticsReconstruction statistics
Nephthytis Poinsettia Schefflera Indoor tree
# image 35 35 40 45
# FG pts 53,000 83,000 43,000 31,000
# leaves 30 ≈ 120 ≈ 450 ≈ 1500
# UAL 6 21 69 35
Recovered leaves 29 116 374 1036
BET (min) 5 2 15 40
UAL = user assisted leaves, BET = branch edit time
ConclusionsConclusions
• Semi-automatic image-base plant modeling
– Simple capturing
– Realistic shape and texture
• Technical contributions:
– Interactive joint segmentation
– Model-based leaf reconstruction
– Interactive branch editing
Image-based Image-based TreeTree Modeling Modeling
Ping Tan, Gang Zeng *, Lu Yuan, Jingdong Wang, Sing Bing Kang, Long Quan
The Hong Kong University of Science and Technology* Microsoft Research
Branch recoveryBranch recovery
• Reconstruction of visible branches
Graph construction
Conversion of sub-graph into branches
User interface for branch refinement
• Reconstruction of occluded branches
Unconstrained growth
Constrained growth
Leaves reconstructionLeaves reconstruction
• Mean shift filtering
• Region split or merge
• Color-based clustering
• User interaction
Adding leaves to branchesAdding leaves to branches
• Create leaves from segmentation
• Synthesizing missing leaves
Approaches to plant modelingApproaches to plant modeling
• Rule-based
– Geometric rules [Weber&Penn 1995]
– L-system [Prusinkiewicz et al. 1994] [Noser et al. 01]
– Botanical rules [De Reffye et al. 1988]
• Image-based
– Volumetric [Shlyakhter et al. 2001] [Reche et al. 2004]
– Statistical [Han et al. 2003]
• Advantages:
– Impressive-looking plants, trees, and forests
• Disadvantages:
– Difficult to use for non-expert
– Difficult to exactly match appearance of actual plants
Rule-based plant modelingRule-based plant modeling
[Weber&Penn 1995]
[Prusinkiewicz et al. 1994]
[Phillippe De Reffye et al. 1988]
• Advantages:
– Details of real plant are captured in image
• Disadvantages:
– Limited realism (visual hull)
– Not manipulable (volumetric representation)
Image-based plant modeling Image-based plant modeling
[Reche et al. 2004]
[Shlyakhter et al. 2001]
[Han et al. 2003]