estimating natural illumination from a single outdoor scene final

Estimating natural illumination from a single outdoor image12/8/09

Estimating natural illumination from a single outdoor scene

Sagnik Dhar (107219686)Debaleena Chattopadhay (107224101)

Course Project for CSE – 591Fall ‘09

Based on, “Estimating Natural Illumination from a Single Outdoor Image” by Jean-

Francois Lalonde, Alexei A. Efros, and Srinivasa G. NarasimhanCVPR ‘09

Estimating natural illumination from a single outdoor image12/8/09

Estimation of illumination

Estimating natural illumination from a single outdoor image12/8/09

Why do people care about illumination ?

• Insertion of objects

• Retrieval of objects from a database

Estimating natural illumination from a single outdoor image12/8/09

Approach

Natural Outdoor Daytime Scene

Extract Plausi-ble Cues

Output a Probable Distribution of Sun’s

Zenith angle

Sky Cue

Shadow Cue

Perez Modeling of

Skies

Simulate Normal

Distribution of sun positions

Determining Best Linear

Shadow

Vertical Surface Cue

Intensity Gradi-ents Vote For Pos-

sible Sun positions

Intensity Of The Vertical Surfaces

Weighed By Their Size Vote For Possible

Sun Directions

Compute Size Ratio Of Surfaces

To The Scene

Estimating natural illumination from a single outdoor image12/8/09

Sky Cue

Determine Horizon Line Extract Sky Segment

Classify into Clear, Patchy or Overcast

category and Process(KNN)

Simulate Perez Sky Model to create 360 skies (each

for a sun zenith angle)(Video)

Taking our sky segment as the mean, find a probability

distribution of sun zenith angle

Estimating natural illumination from a single outdoor image12/8/09

KNN Classification of skies

Sky

Clear Patchy Overcast

Binary segmentation to remove clouds Discard Cue

Continue

Estimating natural illumination from a single outdoor image12/8/09

Shadow Cue

Extract ground Segment

Detect Longest Linear Shadow

Shadow pixel intensities vote for

probable sun directions

Convert to CIE L*a*b Color

Space

Detect Shadow Pixels in b channel

Find Intensity orientation of the shadow

pixels

Cluster Using K-means

Top Cluster votes for Sun

Directions

Estimating natural illumination from a single outdoor image12/8/09

Vertical Surface Cue

Extract Vertical Surfaces

Compute surface size proportionality with respect to the scene.

Find intensity of each surfaces

Surface size weighted by intensity vote for their preferred Sun direction

Estimating natural illumination from a single outdoor image12/8/09

Sun Zenith angle

Estimating natural illumination from a single outdoor image12/8/09

Cue Combination

1ST

2ND3RD

4TH

271°Sky

14°Shadow

BehindVertical Surfaces

Estimating natural illumination from a single outdoor image12/8/09

Results- Best Case

Estimating natural illumination from a single outdoor image12/8/09

Results- Best Case

Estimating natural illumination from a single outdoor image12/8/09

Results- Best Case

1ST

2ND3RD

4TH

100°Sky

8°Shadow

RightVertical Surfaces

Estimating natural illumination from a single outdoor image12/8/09

Results- Best Case

Estimating natural illumination from a single outdoor image12/8/09

Results- Best Case

Estimating natural illumination from a single outdoor image12/8/09

Results- Best Case

1ST

2ND3RD

4TH

181°Sky189°

Shadow

LeftVertical Surfaces

Estimating natural illumination from a single outdoor image12/8/09

Results- Sky + Vertical Cue

Estimating natural illumination from a single outdoor image12/8/09

Results- Sky + Vertical Cue

Estimating natural illumination from a single outdoor image12/8/09

Results- Sky + Vertical Cue

1ST

2ND3RD

4TH

271°Sky

BehindVertical Surfaces

Estimating natural illumination from a single outdoor image12/8/09

Results- Shadow + Vertical Cue

Estimating natural illumination from a single outdoor image12/8/09

Results- Shadow + Vertical Cue

Estimating natural illumination from a single outdoor image12/8/09

Results- Shadow + Vertical Cue

1ST

2ND3RD

4TH

14°Shadow

BehindVertical Surfaces

Estimating natural illumination from a single outdoor image12/8/09

Results- Shadow + Vertical Cue

Estimating natural illumination from a single outdoor image12/8/09

Results- Shadow + Vertical Cue

Estimating natural illumination from a single outdoor image12/8/09

Results- Shadow + Vertical Cue

1ST

2ND3RD

4TH

9°Shadow

BehindVertical Surfaces

Estimating natural illumination from a single outdoor image12/8/09

Results- Failure Cases

Estimating natural illumination from a single outdoor image12/8/09

Results- Failure Cases

Estimating natural illumination from a single outdoor image12/8/09

Results- Failure Cases

1ST

2ND3RD

4TH

5°Shadow

BehindVertical Surfaces

292°Sky

Estimating natural illumination from a single outdoor image12/8/09

Results- Failure Cases

Estimating natural illumination from a single outdoor image12/8/09

Results- Failure Cases

Estimating natural illumination from a single outdoor image12/8/09

Results- Failure Cases

1ST

2ND3RD

4TH

16°Shadow

BehindVertical Surfaces

0°Sky

Estimating natural illumination from a single outdoor image12/8/09

Human Validation

No. of Humans: 3

No. of Images: 58

Human Consensus

Human Contradiction

Accuracy

27 31 46%

Match with Human

Mismatch Accuracy

22 36 37%

Estimating natural illumination from a single outdoor image12/8/09

Cue Weightage Investigation

Estimating natural illumination from a single outdoor image12/8/09

Conclusion

• Accurate estimation of sun position – Tough problem for human visual system too.

• The photographic eye doesn’t bother to capture “informative” cues for visual systems to analyze.

• The weak cues are strongly coupled and highly susceptible to incorrect conclusion.

• Shadow detection from a single image is still in the need of better algorithms.

Estimating natural illumination from a single outdoor image12/8/09

Thank You

To end, a quote I came across recently,

“All I see is the sun; And even when it's hiding behind a smoke of cloud, the shadow tells me all that there is...”