MOBISPEC: DRAAGBARE (HYPER)SPECTRALE CAMERA’S VOOR OP DE WERKVLOER

KICK-OFF IFAST

16-03-2016

DR. ANDY LAMBRECHTS (I2 - IMEC)

CAROLINA BLANCH (I2 – IMEC)

200mm

imager

line Silicon

solar

cell line

Organic

solar

cell lineNERF

lab

300mm

pilot

line

Nano

Bio labs

450mm

ready

▸ World-leading research in nano-

electronics

▸ >1B€ cumulated investment on-site in

leading-edge semiconductor

factories

▸ Scientific knowledge with innovative

power of global partnerships in ICT,

healthcare and energy

▸ Focus on industry-relevant

technology solutions

▸ International top talent in a

unique high-tech environmentcommitted to provide building blocks

for a better life in a sustainable society

IMEC MISSION

Belgium

Netherlands

Taiwan

office Japanoffice USA

China

India

▸ Established by state

government of Flanders in

1984

▸ Independent non-profit

organization

▸ Worldwide reach

▸ Initial investment: 62M€

▸ Initial staff: ~70

HISTORY

▸ Total revenues in 2015 of 415M€,

growth of 5%

- 80% industry vs. 20% funded

▸ 132 patents awarded & 133

patents submitted in 2011

▸ Collaboration with 600 companies

& 208 universities

▸ 600 residents on-site from

industrial partners

FACTS & FIGURES

STATE-OF-THE-ART

Research to Production FACILITIES

200mm

line

Silicon

solar cell

line

Organic

solar cell

line NERF

lab

300mm

pilot line

Nano

biolabs

450mm

ready

Foundries Fablite FablessLogic IDMMemory IDM

GLOBAL NETWORK OF SEMICONDUCTOR PARTNERS

Green Radio

Low power RF communications

ImagingImage sensors

& Vision systemsMicrofluidicBody AreaNetworks

Life-SciencesEnergyPhotovoltaic

Power devicesLEDs

Sensor systems for industrial applications

IMEC ECO-SYSTEM FOR NANOTECHOrganicelectronics

GO OPTICAL

Si PHOTONICS

Core CMOS (22nm and below)

Lithography Devices Interconnects

CMORE

Photonic, MEMS,

3D integration

ImagingImage sensors

& Vision systems

© IMEC 2015

Imec Imaging team focus and scope

Materials & Fab link

HW Design (Analog + pixels +

post)

Camera & System Design

Low level image proc.

(LFI, HS) image proc.

Application SW

Full system feasibility

imec customer

Large data sets and services

9© IMEC/KU Leuven 2016

OVERVIEW

What is Hyperspectral Imaging?

Imec sensor technology

Portable and compact hyperspectral devices

Validation of our Mosaic sensors

Challenges

Conclusions

12© IMEC/KU Leuven 2016

Hig

h d

iscr

imin

atio

n p

ow

er

Single point

with highest

spectral

resolution

SPECTROSCOPY

CONVERGENCE OF IMAGING & SPECTROSCOPY

High speed, high spatial resolutionIMAGING

RGB filters

on multiple pixelsIMAGING

SPECTROSCOPY

IMEC approach: spectral filters on multiple pixels

Tiled patternMosaic pattern OR OR Line pattern

13© IMEC/KU Leuven 2016

SPECTRAL FILTERS APPLIED TO IMAGE SENSOR WAFERS BY SEMICONDUCTOR PROCESS

IMEC hyperspectral filter structures processed at wafer-level on top of commercial CMOS

image sensor wafer (here on CMOSIS’s CMV2000 & CMV4000 sensors)

14© IMEC/KU Leuven 2016

Ximea

NEW EVALUATION KITS BASED ON USB3.0 CAMERA

465nm

474nm

485nm

496nm

510nm

522nm

534nm

546nm

562nm

578nm

586nm

600nm

608nm

624nm

630nm

548nm

15© IMEC/KU Leuven 2016

SNAPSHOT MOSAIC HSI SENSOR (4X4 VIS)

■ Key specifications■ Spectral resolution: 4x4 mosaic (1filter / pixel) = 16bands in 470-630nm

■ FWHM: ~ 15nm

■ Spatial resolution: from 512x272 (RAW per band)

■ Speed: up to 340 data-cubes / s (max sensor limit)

2048 pixels

1088

pix

els

CMOSIS CMV2000474

nm

465

nm630

nm

624

nm

608

nm

485

nm

4x4 mosaic = 16 bands

496

nm

510

nm

522

nm

534

nm

546

nm

586

nm

578

nm

562

nm

548

nm

600

nm

16© IMEC/KU Leuven 2016

SNAPSHOT MOSAIC HSI SENSOR (5X5 NIR)

■ Key specifications■ Spectral resolution: 5x5 mosaic (1filter / pixel) = 25bands in 600-1000nm

■ FWHM: ~ 10-15nm

■ Spatial resolution: from 410x218 (RAW per band)

■ Speed: up to 340 data-cubes / s (max sensor limit)

2048 pixels

1088

pix

els

CMOSIS CMV2000

5x5 mosaic = 25 bands

835

nm

915

nm960

nm

890

nm

805

nm

760

nm

675

nm

700

nm

775

nm

850

nm

930

nm

945

nm

875

nm

715

nm

730

nm

600

nm615

nm

630

nm

645

nm660

nm

745

nm

820

nm

900

nm

975

nm

790

nm

18© IMEC/KU Leuven 2016

ROADMAP FOR OFF-THE-SHELF HSI SENSORS

2012 2013 2014 2015 2016+

Snapshot

‘RGB+NIR’

LINESCAN

‘Wedge’

GEN1 600-975nm

(100+bands)GEN2 450-960nm

(150+bands)

GEN3 420-960nm

(180+ bands)

NIR VNIR VNIR

Snapshot

‘TILED’

GEN1 600-975nm (32bands)

NIR

Snapshot

‘MOSAIC’

GEN3 460-975nm

(40+bands)

VNIR

VIS 460-640nm

(4x4 =16bands)

VIS

NIR 600-975nm

(5x5 = 25bands)

NIR

GEN1 1-1.7um

(100+ bands)

SWIR

19© IMEC/KU Leuven 2016

DUAL CAMERA SENSOR COMBINING VIS & NIR RANGES

First prototype of dual camera ready!

Combining 16 VIS bands (470-630nm) with 25 NIR bands (600-1000nm)

41 bands VIS & NIR images

20© IMEC/KU Leuven 2016

Ximea

bringing hyperspectral

imaging ...

... to machine vision

industry!

21© IMEC/KU Leuven 2016

portable | compact | ready-to-go!

22© IMEC/KU Leuven 2016

Adimec

Tattile

PhotonFocus

XimeaBayspec

3D-One

VRMAGIC

Cubert

Flexible

camera platform Giga/Ethernet interface

Focus on remote

sensing, UAV &

machine vision

MULTIPLE PARTNERS FOR HSI SENSOR CAMERA INTEGRATION

High speed camera Cameralink or CoaXPress interface

Compact optical design with ZIFF socket

to swap different HSI sensors

Robust camera implementation for use in

industrial, medical and global security

Smart camera Giga/Ethernet interface

Reprogrammable FPGA

pre-processing hypercube data inside

camera

Focus on industrial machine vision

Handheld mobile platform First proof-of-concept of IMEC’s

HSI sensors integration to mobile

Android OS open platform

USB2.0 interfaceUSB 3.0 vision camera High-speed USB3.0 camera

Ultra compact camera:

26.4 x 26.4 x 21.2 mm

27grams weight only!

Intelligent vision module Giga/Ethernet interface

Motorized optics

Embedded HSI data processing

include storage

Multi-camera head optionSmart camera Giga/Ethernet interface

FPGA pre-processing

Agriculture, food sorting and

industrial applications

23© IMEC/KU Leuven 2016

HSI SOFTWARE IMAGE PROCESSING PIPELINE

Camera/Sensor

SpecificApplication Specific

Pre-Processing

Camera calibration, registration,

synchronization, reflectance

calculation...etc

Feature

Extraction &

Selection

Feature

Classification/

Recognition

Spectral un-mixing,

smoothening, PCA, SVM, k-

means clustering, ...etc

Post-Processing

Visualization, GUI... etc

HSI

‘cube’

Application Specific (to be developed

for each platform)

IMEC can support these

SOFTWARE / Application layers

... integration path into UAVs for

precision agriculture

Cropping area under the Field Phenotyping Platform (FIP) of the Group of Crop Science

at the ETH Research Station for Plant Sciences Lindau-Eschikon, ETH Zürich

APPLICATION

● Wheat phenotyping in

Switzerland

RESULTS

● Identification and

diagnostics of crop traits

BENEFITS

● Non-destructive analysis

● Diagnostics of

● Stress

● Disease

● Weeds

● Nutrients

RGB | Hyperspectral (Snapshot mosaic)

27© IMEC/KU Leuven 2016

snapshot hyperspectral

imaging in medical

28© IMEC/KU Leuven 2016

REAL-TIME OXYGENATION MAPS

View live video here https://vimeo.com/132097972

SNm VIS

Reconstructed Hb/HbO/HBT maps extracted from HSI snapshot mosaic

4x4 = 16bands HSI sensor on epileptic patient during brain surgery

29© IMEC/KU Leuven 2016

POTENTIAL BROUGHT TO FOOD/SORTING INDUSTRY

■ Fast and automatic online monitoring

■ Optimization of process control

■ Fast and robust on-site inspection

■ Food quality evaluation/grading

■ Food composition control

■ Detection of allergens

■ Detection of foreign objects

Industrial

Consumer

30© IMEC/KU Leuven 2016

VALIDATION OF MOSAIC NIR CAMERA: PREDICTION OF ALKALINE DEGREE IN BONE SAMPLES

10 15 20 25 30 35 40 45 505

10

15

20

25

30

35

40

45

50

55

Measured values

Pre

dic

ted

va

lues

Thawed Beef samples

Mosaic NIR

(675-975nm)

25 bandsRed line = average prediction

Example: Good prediction of alkaline

degree achieved based on HSI

Acid treatment

Further processing

Alkaline treatment

HSI provides

feedback on acid

and alkaline

degrees

Hyperspectral Imaging for

process control optimization

31© IMEC/KU Leuven 2016

HIGH SPEED ANOMALIES DETECTION WITH

25+ BANDS SNAPSHOT MOSAIC SENSOR

View live video here https://vimeo.com/132098267

SNm NIR

real leaf

plastic

leaf

32© IMEC/KU Leuven 2016

FOOD FRESHNESS MONITORING FOR FOOD

QUALITY GRADING APPLICATIONS

■ 100% inspection via gill’s blood spectrum

■ Determine food quality freshness within few hours till 4 days

33© IMEC/KU Leuven 2016

CHALLENGES IN ON-SITE HSI INSPECTION

Controlled and stable portable illumination

Calibration for measurements under dynamic

conditions

Dual sensor with VIS and NIR ranges jointly covered

Real-time classification & analysis

Relevant wavelength selection when required

34© IMEC/KU Leuven 2016

MOBISPEC PARTICIPANTS

35© IMEC/KU Leuven 2016

CONCLUSIONS

Imec technology enables compact and fast cameras

for hyperspectral imaging

Challenges to be solved for robust on-site inspection

with portable devices

High potential for applications using hyperspectral

portable devices in food industry

On-site food quality evaluation/grading

Food/material composition control/ chemical analysis

Foreign object/ allergen detection

36© IMEC/KU Leuven 2016

THANKS FOR YOUR ATTENTION!