CMU SCS

Indexing and Mining Biological Images

Christos Faloutsos

CMU

UCSB06 2

CMU SCS

THANKS

UCSB06 3

CMU SCS

Outline

• PART1: ViVo: Visual Vocabulary for cat retina images

• [PART2: other related work– FALCON: relevance feedback for image by

content– Drosophila embryo image mining

]

UCSB06 4

CMU SCS

PART1: ViVo

• with Ambuj Singh, Mark Verardo, Vebjorn Ljosa, Arnab Bhattacharya (UCSB)

• Jia-Yu Tim Pan, HJ Yang (CMU)

UCSB06 5

CMU SCS

Detachment Development

Normal1 day after detachment

3 days after detachment

7 days after detachment

28 days after detachment

3 months after detachment

UCSB06 6

CMU SCS

Data and Problem

• (Data) Retinal images taken from cats• (Problem) What happens in retina after

detachment?– What tissues (regions) are involved? – How do they change over time?

• How will a program convey this info?• More than classification

“we want to learn what classifier learned”

UCSB06 7

CMU SCS

Main idea

• extract characteristic visual ‘words’

• Equivalent to characteristic keywords, in a collection of text documents

UCSB06 8

CMU SCS

Visual Vocabulary (ViVo) generation

Step 1: Tile image

Step 2: Extract tile features

Step 3: ViVo

generation

Visualvocabulary

V1

V2

Feature 1

Fea

ture

2

8x12 tiles

UCSB06 9

CMU SCS

ViVos

skip

UCSB06 10

CMU SCS

Evaluation of ViVo method

• how meaningful are the discovered ViVos?

• can they help in classification?

• generality?

• how else can they help biologists create hypotheses?

UCSB06 11

CMU SCS

Biological interpretationID ViVo Description Condition

V1 GFAP in inner retina (Müller cells) Healthy

V10 Healthy outer segments of rod photoreceptors

Healthy

V8 Redistribution of rod opsin into cell bodies of rod photoreceptors

Detached

V11 Co-occurring processes: Müller cell hypertrophy and rod opsin redistribution

Detached

UCSB06 12

CMU SCS

Goals:

• how meaningful are the discovered ViVos?

• can they help in classification?

• generality?

• how else can they help biologists create hypotheses?

UCSB06 13

CMU SCS

Quality of ViVo – by classification

N 1d 3d 7d 28d 28dr 6dO2 3m

N 7 2

1d 7

3d 12 1 1 1

7d 1 8 2

28d 1 23 2

28dr 1 21

6dO2 1 1 9

3m 5

Truth

Predicted

86% accuracy46 ViVos (90% energy)

UCSB06 14

CMU SCS

Goals:

• how meaningful are the discovered ViVos?

• can they help in classification?

• generality?

• how else can they help biologists create hypotheses?

UCSB06 15

CMU SCS

ViVos for protein images

UCSB06 16

CMU SCS

Protein images – MPEG7 CS

Giantin Hoechst LAMP2 NOP4 Tubulin

Giantin 30

Hoechst 30

LAMP2 50 9 1

NOP4 1 8 2

Tubulin 1 23

Truth

Predicted

84% accuracy4 ViVos (93% energy)1-NN classifier

UCSB06 17

CMU SCS

Evaluation of ViVo method

• how meaningful are the discovered ViVos?

• can they help in classification?

• generality?

• how else can they help biologists create hypotheses? ‘ViVo-annotation’!

UCSB06 18

CMU SCS

Automatic ViVo-annotation of images

• A tile represents a ViVo vk if the largest coefficient of the tile is along the kth basis vector

• A ViVo vk represents a class ci if the majority of its tiles are in that class

• For each image, the representative ViVos for the class are automatically highlighted

UCSB06 19

CMU SCS

Which tissue is significant on 7-day?

UCSB06 20

CMU SCS

6 days after O2 treatment

UCSB06 21

CMU SCS

28 days after surgery

UCSB06 22

CMU SCS

Conclusions:

• how meaningful are the discovered ViVos?

• can they help in classification?

• generality?

• how else can they help biologists create hypotheses?

UCSB06 23

CMU SCS

Outcome/status

• ViVos: Automatic Visual Vocabulary generation for biomedical image mining, Bhattacharya, Ljosa, Pan, Verardo, Yang, Faloutsos, Singh; ICDM’05 (one of 5 best student paper award)

• Software – MATLAB code

• Tutorial in SIGMOD’05 (Murphy+Faloutsos)

UCSB06 24

CMU SCS

Outline

• PART1: ViVo: Visual Vocabulary for cat retina images

• PART2: FALCON: relevance feedback for image by content: SEE DEMO, later

• Ongoing work: Drosophila Fly Embryos

UCSB06 25

CMU SCS

FALCON - Example query:Inverted VsVs

Trader wants only ‘unstable’ stocks

UCSB06 26

CMU SCS

Outline

• PART1: ViVo: Visual Vocabulary for cat retina images

• PART2: FALCON: relevance feedback for image by content: SEE DEMO, later

• Ongoing work: Drosophila Fly Embryos

UCSB06 27

CMU SCS

FEMine: Mining Fly Embryos

UCSB06 28

CMU SCS

FEMine: Mining Fly Embryos

UCSB06 29

CMU SCS

FEMine: Drosophila embryos

• Feature extraction

• ICA

• query by image content, mining, clustering

with Andre Balan, Eric Xing, Tim Pan

UCSB06 30

CMU SCS

Conclusions

Machine vision + Data mining + Data bases + Biology:

=> necessary partners