Optical Neural System Optical Neural System Imaging SurveyImaging Survey

November 15, 1999November 15, 1999Andreas G. NowatzykAndreas G. Nowatzyk

OutlineOutline

• Background and Motivation

• System Overview

• Backscatter Imager

• Fluorescent Imager

• Microtome

• Staining Unit

• Computational Aspects

• Summary

Context• Enabling technologies

– Moore’s Law: Billions of Billions of cycles– Interconnect networks: Multi-Gbytes/s I/O– Optics: confocal microscopy, lasers– Biochemistry: selective staining– Computer science: Image processing– Mechanics: Chip fabrication technologies

• Interdisciplinary research: Combining all these new tools to open up new research areas

Ultimate Goal: Scan a Mouse in Ultimate Goal: Scan a Mouse in 3D3D• Eventually, Eventually,

reverse reverse engineer a engineer a mouse brain, mouse brain, including the including the entire rodent entire rodent nervous nervous systemsystem

Computational ChallengeComputational Challenge

• 2 x 2 x 4 cm2 x 2 x 4 cm3 3 specimen volume specimen volume

• 0.2 0.2 m resolutionm resolution

• = 100,000 x 100,000 x 200,000 = 100,000 x 100,000 x 200,000 voxelsvoxels

• 6 x 12 bit per voxel6 x 12 bit per voxel

• 16,763,806 Gbytes of raw data16,763,806 Gbytes of raw data

• 200:1 compression (lossy)200:1 compression (lossy)

• => 1700 tapes (8mm, 50Gbytes/tape)=> 1700 tapes (8mm, 50Gbytes/tape)

Computational ChallengeComputational Challenge

• 20 Msamples/s per channel (12 bits 20 Msamples/s per channel (12 bits each)each)

• 6 x 64 channels6 x 64 channels

• ~50% scan duty cycle~50% scan duty cycle

• 2x for complex phase sensing2x for complex phase sensing

• = 11,520 Mbytes/s average data = 11,520 Mbytes/s average data raterate

System OverviewSystem Overview

Basic approach, instrument Basic approach, instrument architecture, and computing architecture, and computing

infrastructureinfrastructure

Functional ImagingFunctional Imaging

• Selective labeling with Fluorescent Selective labeling with Fluorescent DyesDyes

• Genetically engineered Labels (GFP)Genetically engineered Labels (GFP)

VS.

Basic ApproachBasic Approach

• Use of light microscopy plus selective Use of light microscopy plus selective functional imagingfunctional imaging

• Five step processFive step process– Bulk imaging into freshly cut sampleBulk imaging into freshly cut sample

– Mechanical sectioning via integrated Mechanical sectioning via integrated microtomemicrotome

– Automated, continuous stainingAutomated, continuous staining

– Functional, fluorescent imagingFunctional, fluorescent imaging

– Data fusion, compression and archival Data fusion, compression and archival storagestorage

Confocal Light MicroscopyConfocal Light Microscopy

• Using one objective lens twiceUsing one objective lens twice

• Point-spread function squaredPoint-spread function squared

Instrument OverviewInstrument Overview

BackscatterImager

Staining Unit

SpecimenHolder

Transfer Unit

Carrier Tape

Microtom

FluorescentImager

Backscatter ImagerBackscatter Imager

Optical system, mechanical Optical system, mechanical components, scan pathcomponents, scan path

Design ObjectivesDesign Objectives

• Need to maximize throughputNeed to maximize throughput

• Maximize practical resolutionMaximize practical resolution

• Optical sectioning into the exposed Optical sectioning into the exposed sample surfacesample surface

XZ Detection FunctionXZ Detection Function

High NA (0.9) objective with pinhole High NA (0.9) objective with pinhole detectordetector

PMT

XZ Detection FunctionXZ Detection Function

Confocal illuminationConfocal illumination

PMT

XZ Detection FunctionXZ Detection Function

Michelson interferometerMichelson interferometer

PMT

Acousto-Optical ModulatorAcousto-Optical Modulator

• Exploit Doppler shift to split laser Exploit Doppler shift to split laser into two components with differing into two components with differing frequencyfrequency

RF

XZ Detection FunctionXZ Detection Function

Heterodyne detectionHeterodyne detection

PMT

X

f

Heterodyne DetectionHeterodyne Detection

• Elimination of black-level drift through AC-Elimination of black-level drift through AC-coupled amplifierscoupled amplifiers

• Contrast independent of reference beam Contrast independent of reference beam intensity (within dynamic range of detector)intensity (within dynamic range of detector)

• Optical phase transferred to electrical Optical phase transferred to electrical domaindomain

• Simultaneous capture of phase and Simultaneous capture of phase and magnitudemagnitude

• Enables wave-front reconstruction / Enables wave-front reconstruction / holographyholography

Solid State DetectorSolid State Detector

HP SS-detectory = 2.0328x + 24.826y = 2.0469x + 7.2512

y = 2.036x - 13.267

-130

-120

-110

-100

-90

-80

-70

-60

-50

-40

-30

-20

-70 -60 -50 -40 -30 -20

Optical Input Power [dbm]

Sig

nal

Ou

t [d

bm

]

Sig 0db [dbm] Sig -20db [dbm] Sig -40db [dbm]

Noise floor [dbm/Hz] PMT(700V),0db PMT(700V),-20db

PMT(700V), -40db PMT(700V) noise floor Linear (Sig 0db [dbm])

Linear (Sig -20db [dbm]) Linear (Sig -40db [dbm])

• Better quantum efficiency (~ 85 vs Better quantum efficiency (~ 85 vs 30 %)30 %)

• Better dynamic range (> 120 db)Better dynamic range (> 120 db)

Qualitative VerificationQualitative Verification

First TestsFirst Tests

Z-Axis Selection Vs. BandwidthZ-Axis Selection Vs. Bandwidth

Backscatter Imager SummaryBackscatter Imager Summary

• Optimized for single purposeOptimized for single purpose– less relay optic, no eye piece, no turret, etc.less relay optic, no eye piece, no turret, etc.

• Scan system optimized for bulk Scan system optimized for bulk operationoperation

• Heterodyne detection for improved Z-Heterodyne detection for improved Z-axis resolution (full sampling of the axis resolution (full sampling of the optical phase)optical phase)

• Multiple wavelengthsMultiple wavelengths

• Requires computer controlled alignmentRequires computer controlled alignment

Fluorescent ImagerFluorescent Imager

Optical system, mechanical Optical system, mechanical components, coherent detectioncomponents, coherent detection

Design ObjectivesDesign Objectives

• Need to match throughput of Need to match throughput of backscatter imagerbackscatter imager

• Film based input media, accessible Film based input media, accessible from both sides with symmetric, from both sides with symmetric, optical propertiesoptical properties

• Support for optical sectioningSupport for optical sectioning

ProblemsProblems

• Dye saturationDye saturation

• Signal to noise ratioSignal to noise ratio

• Incoherent signalIncoherent signal

Need to maximize light gathering Need to maximize light gathering ability and quantum efficiency of ability and quantum efficiency of detectordetector

Scan SystemScan System

• Sorry, the few slides were removed Sorry, the few slides were removed due to pending (but incomplete) due to pending (but incomplete) patent applicationspatent applications

MicrotomMicrotom

Functions, mechanical Functions, mechanical componentscomponents

Design ObjectivesDesign Objectives

• Integrated with optical systemIntegrated with optical system

• Fully automaticFully automatic

• Precise control of cutting plane and Precise control of cutting plane and related parametersrelated parameters

Transferring fragile Objects to Transferring fragile Objects to FilmFilm• Microtom with stationary knifeMicrotom with stationary knife• Floating pick-upFloating pick-up• Match surface velocity to cutting Match surface velocity to cutting

speedspeed• Minimize surface tensionMinimize surface tension• Electrostatic transferElectrostatic transfer

Knife AssemblyKnife Assembly

6.292

4.694

3.017

far end of

cutting area

1.300

5.700

5.200

2.700

1.82 in.2.051

2.103

1.300

1.719

Instrument BaseInstrument Base

Microscope EnvironmentMicroscope Environment

• -20 to -30 degree C, controlled -20 to -30 degree C, controlled operating temperatureoperating temperature

• dust free, controlled flow, dry dust free, controlled flow, dry nitrogen atmospherenitrogen atmosphere

• vibration isolationvibration isolation

• issue: sublimation (prevent by index issue: sublimation (prevent by index matching fluid)matching fluid)

Mechanical componentsMechanical components

• Linear air bearingsLinear air bearings

• Voice-coil motor direct driveVoice-coil motor direct drive

• Laser interferometer position Laser interferometer position sensing with 10nm resolutionsensing with 10nm resolution

• Piezo-actuators for knife positioningPiezo-actuators for knife positioning

Specimen chamberSpecimen chamber10.04 in.

3.69 in.

Staining UnitStaining Unit

Outline, ChallengesOutline, Challenges

Film as Sample Slice CarrierFilm as Sample Slice Carrier

• Candidate: DuPont Cronar 410 Candidate: DuPont Cronar 410 polyester flim, gelatin coated, polyester flim, gelatin coated, 100100mm

• Need to maintain sample adhesion Need to maintain sample adhesion during stating processduring stating process

• Optically clear, substitute for cover-Optically clear, substitute for cover-slips (need to be witting correction slips (need to be witting correction range)range)

• Chemically inertChemically inert

Continuous Staining ProcessContinuous Staining Process

• Tight control of process parameters Tight control of process parameters (temperature, flow rate, chemical (temperature, flow rate, chemical concentrations, etc.)concentrations, etc.)

• Clean-room environment: dust-free, Clean-room environment: dust-free, high purity, no manual stepshigh purity, no manual steps

• Uniform, predictable distortionsUniform, predictable distortions

Computational AspectsComputational Aspects

Where does all the money go?Where does all the money go?

Computational InfrastructureComputational Infrastructure

• Instrument controlInstrument control

• Front-end signal processingFront-end signal processing

Control System OverviewControl System Overview

• Linux basedLinux based

• 3 Functional 3 Functional Blocks:Blocks:– System ControlerSystem Controler

– Signal ProcessingSignal Processing

– File system / File system / DatabaseDatabase

Image processingImage processing

• Deconvolution (CAT scan, MRI)Deconvolution (CAT scan, MRI)• Super-Resolution [Cheeseman et al]Super-Resolution [Cheeseman et al]

• 3D reconstruction3D reconstruction

• Image fusionImage fusion

• 3D Compression3D Compression

Circuit Extraction AlgorithmCircuit Extraction Algorithm

• Linear scan of the data setLinear scan of the data set• Connectivity function integrated Connectivity function integrated

with Bayesian super-resolution with Bayesian super-resolution algorithmalgorithm

Summary of Instrument Summary of Instrument CapabilitiesCapabilities• 6 band (3 backscatter + 3 6 band (3 backscatter + 3

fluorescent labeled) confocal fluorescent labeled) confocal microscopemicroscope

• 0.20.2m (or better) 3D resolutionm (or better) 3D resolution

• Integrated sectioning and stainingIntegrated sectioning and staining

• Image acquisition and processing Image acquisition and processing speed sufficient for large volume speed sufficient for large volume scanningscanning

SummarySummary

• Large scale, high resolution scanning of Large scale, high resolution scanning of biological specimen will become biological specimen will become practicalpractical

• Automated tracing of neurological Automated tracing of neurological systems is conceivablesystems is conceivable

• Provides a clear, long term research Provides a clear, long term research focusfocus

• Significant Research potentialSignificant Research potential

• Intermediate, practical spin-off potentialIntermediate, practical spin-off potential