Roles of local and global information in image classification

Devi Parikh

Coast Highway Mountain Street

Forest Inside city Open country Tall buildings

Bathroom Bedroom Dining room Gym

Kitchen Living room Movie theater Stair case

Aeroplane Car-rear Face

Ketch Motorbike Watch

OSR

ISR

CAL

2

Jumbled Images

3

Human Study: Intact Images

Human Study: Jumbled Images

Human Recognition of Jumbled Images

OSR ISR CAL0

10

20

30

40

50

60

70

80

90

100

Intact

JumbledAccu

racy

6

How do we recognize them?

Majority Vote of Local Independent

Decisions?

MRF Inference?

…

Are existing machine implementations of this model good enough?

7

Human Studies: Individual Blocks

8

Human Recognition of Individual Blocks

OSR ISR CAL25

30

35

40

45

50

Accu

racy

9

Machine Experiments

• Assign blocks to codewords

• Accumulate p(scene|codewords)

• Pick class with highest votes

Majority Vote ResultsPr

opor

tion

of m

atch

ed re

spon

ses

OSR ISR CAL0

10

20

30

40

50

60

70

80

90

100

Random

Human Most Confident Block

Human Majority-Vote

Machine Majority-Vote

Inter-Human Agreement

11

Classification of Jumbled Images

OSR ISR CAL0

10

20

30

40

50

60

70

80

90

100

HumanMachine

Accu

racy

Need advanced modeling of global information12