optical microscopy with flexible axial capabilities using a...

Journal of Microscopy, Vol. 258, Issue 3 2015, pp. 212–222 doi: 10.1111/jmi.12235

Received 29 August 2014; accepted 23 January 2015

Optical microscopy with flexible axial capabilities usinga vari-focus liquid lens

Y U F U Q U & H A I J U A N Y A N GSchool of Instrument Science and Opto-Electronics Engineering, Beihang University, Beijing, China

Key Laboratory of Precision Opto-Mechatronics Technology of Education Ministry, Beijing, China

Key words. Autofocusing, extended depth of field, flexible axial capabilities,three-dimensional reconstruction.

Summary

The axial imaging range of optical microscopy is restricted byits fixed working plane and limited depth of field. In this paper,the axial capabilities of an off-the-shelf microscope is improvedby inserting a liquid lens, which can be controlled by a drivingelectrical voltage, into the optical path of the microscope. First,the numerical formulas of the working distance and the mag-nification with the variation of the focus of the liquid lens areinferred using a ray tracing method and conclusion is obtainedthat the best position for inserting a liquid lens with consistentmagnification is the aperture plane and the rear focal planeof the objective lens. Second, with the liquid lens embedded inthe microscope, the numerical relationship between the mag-nification and the working distance of the proposed flexible-axial-capability microscope and the liquid lens driving voltageis calibrated and fitted using the inferred numerical formu-las. Third, techniques including autofocus, extending depth offield and three-dimensional imaging are researched and ap-plied, improving the designed microscope to not only flexiblycontrol its working distance, but also to extend the depth offield near the variable working plane. Experiments show thatthe presented flexible-axial-capability microscope has a longworking distance range of 8 mm, and by calibrating the mag-nification curve within the working distance range, samplescan be observed and measured precisely. The depth of field canbe extended to 400 μm from the variable working plane andis 20 times that of the off-the-shelf microscope.

Introduction

The axial imaging range of an optical microscope is restrictedby its fixed working plane and limited depth of field. Whenobserving samples with a larger depth range such as living

Address: No.37 XueYuan Road, HaiDian District, BeiJing, 100191, CHINACorrespondence to: Yufu Qu, School of Instrument Science and Opto-Electronics

Engineering, Beihang University, Beijing 100191, China. Tel: 86-10-82317336;

fax: 86-10-82317336; e-mail: [email protected] or [email protected]

cells, spherical surfaces or titled surfaces, the axial position ofsamples or microscopic objectives has to be adjusted manually,which increases the complexity and work required for operat-ing a microscope, and limits the reproducibility of results. Forthe past few decades, many methods have been proposed to de-velop microscopes capable of flexible axial observation. Thesemethods can be classified into three categories: first, autofocus(AF), which facilitates the acquisition of clear sample imagesby adjusting the objective or sample to the work distance of theoptical microscope; second, Extended Depth of Field (EDOF),which increases the range of axial depth for clear imaging nearthe work distance; third, three-dimensional imaging (3DI)which helps in rendering three-dimensional (3D) models ofsamples. AF methods (Sun et al., 2004, 2005) include activeand passive AF. Active AF utilizes an active distance measur-ing element to measure the objective–sample distance, thenadjusts the distance to acquire a clear image. Passive AF uti-lizes an image-processing algorithm (Liang & Qu, 2012; Wuet al., 2012) to evaluate the focus level of images, which is thenfed back to a motion control system to adjust the optical compo-nents and acquire a clear image. EDOF has been a research fo-cus for the past few years and many strategies (Qu et al., 2012)have been proposed such as those incorporating multifocus de-tectors, moving detectors, variable apertures, mask or a phaseplate, spherical aberration or chromatic aberration, volumet-ric sampling methods (Liu & Hua, 2011), etc. EDOF helps toextend the imaging range near the working plane of a mi-croscope, however when samples are placed far from the focusplane, clear images cannot be formed. 3DI methods or applica-tions can resolve 3D structures of samples by means of opticalsectioning (Conchello & Lichtman, 2005), 3D imaging withaxially distributed sensing (Schulein et al., 2009), shape-from-focus (SFF) algorithms (Nayar, 1992), light-field microscopy(Levoy et al., 2006), 3D imaging with positioned lens array(Arai et al., 2013) and digital holographic microscopy (Kemperet al., 2007), some of which are expensive since the devicestructure is sophisticated and complex. However, these threemethods only increase the axial capabilities of a microscope

C© 2015 The AuthorsJournal of Microscopy C© 2015 Royal Microscopical Society

O P T I C A L M I C R O S C O P Y W I T H F L E X I B L E A X I A L C A P A B I L I T I E S 2 1 3

near its working plane, which only extends the axial capabili-ties of a microscope in a limited depth-of-field range.

In this study, to improve the axial capabilities of a micro-scope for flexibility over a long distance and increased depth-of-field range, an axial-capability microscope is designed byinserting a liquid lens into an off-the-shelf microscope. Theposition of the liquid lens is determined by theoretical analysisto achieve a relatively small magnification shift. The flexible-axial-capability microscope can not only flexibly control itsworking distance, but can also extend the depth of field nearthe shifted working plane using a volumetric optical samplingmethod and subsequent image processing. 3D modelling ofsamples and all-in-focus images can be rendered by taking afocal stack while changing the focus of the liquid lens andapplying subsequent algorithms.

The rest of the paper is organized as follows: The optical lay-out of a flexible-axial-capability microscope and the impact ofthe variable focus of the liquid lens on the working distance andthe magnification of a microscope is described in section ‘Op-tical system for flexible-axial-capability microscopy’. Section‘Flexibly controlling the working distance of a microscope’ in-troduces methods that use the liquid lens as a focus element toadjust the working distance of a microscope in order to achieveclear images of samples placed at different distances. In section‘EDOF and 3DI for the flexible-axial-capability microscope’,the EDOF and 3DI methods are described. The experimentalsystem and test results are discussed in section ‘Experimentalsetup and results’. Finally, we conclude our work in section‘Conclusions’.

Optical system for flexible-axial-capability microscopy

The flexible-axial-capability microscope discussed in this pa-per serves two major functions: first, flexibly controlling theworking distance of a microscope by changing the focus ofthe liquid lens, so it is needed that the magnification of the

optical microscope changes minimally when varying the fo-cus plane to observe samples at different depths; second, usingmethods of volumetric optical sampling and focal stacking torender EDOF images. Here, it is necessary to maintain mag-nification in an arbitrary identical plane in the range of theEDOF. Otherwise, EDOF imaging will introduce inconveniencefor subsequent image processing because of the image pointshift caused by inconsistent magnification. In order to achieverelatively consistent magnification discussed above and deter-mine an ideal position of the liquid lens in the optical pathof the microscope, theoretical analysis of the optical path isconducted.

The simplified optical path of the flexible -axial-capabilitymicroscope is shown in Figure 1. The optical layout consistsof an objective lens, a liquid lens, and a CCD (charge-coupleddevice). The objective lens and the liquid lens are assumed to bethin lenses. The focus points of the objective are F1 and F ′

1, andthe focal lengths of the objective are f1 and f ′

1, with f ′1 = − f1.

The focus points of liquid lens are F2 and F ′2, and the focal

lengths are f2 and f ′2, with f ′

2 = − f2. The distance betweenthe objective lens and liquid lens is d . The principle points ofthe optical group consisting of the objective and the liquid lensare H and H ′, and the focal length is f and f ′, with f ′ = − f .The CCD detector is placed at a distance l ′

2 from the liquid lens.Since the aperture of the liquid lens is smaller than the apertureof the objective lens, the aperture of the optical group is locatedin the liquid lens. Suppose that the magnification of the systemwith respect to the conjugate object plane of the CCD whenvarying the focus of the liquid lens is β1, and the magnificationwith a fixed object plane when varying the focus of the liquidlens is β2. Then, according to the magnification formula ofoptical systems, the magnification of the optical group withrespect to the conjugate object plane of the CCD β1 is:

β1 = − x′F

f ′ = 1 + l ′2d

f ′1 f ′

2− l ′

2

f ′2

− l ′2

f ′1

− df ′

1

. (1)

Fig. 1. The simplified optical path of a flexible-axial-capability microscope.

C© 2015 The AuthorsJournal of Microscopy C© 2015 Royal Microscopical Society, 258, 212–222

2 1 4 Y . Q U A N D H . Y A N G

Fig. 2. Magnification of a fixed object depth with different liquid lens foci.

The liquid lens diopter, φ2, is:

φ2 = 1f ′

2. (2)

Then β1 can be represented as:

β1 = 1 − l ′2

f ′1

− df ′

1

+ l ′2

f ′1

(d − f ′

1

)φ2. (3)

In order to achieve relatively constant magnification withthe conjugate object plane when the focus of the liquid lenschanges, the differential relation between the magnificationand the diopter of liquid lens is derived:

∂β1

∂φ2= l ′

2df ′

1

− l ′2 = l ′

2

(d − f ′

1

)f ′

1

. (4)

Let ∂β1∂φ2

= 0, then we get l ′2 = 0 or d = f ′

1 which meansthat β1 does not change when the liquid lens is placed cling-ing to the CCD plane or the rear focal plane of the objective.Since the aperture of the liquid lens (Arctic 39N0, Varioptic,3.5 mm) is smaller than the CCD plane (8.4 mm × 6.4 mm),the image will be occluded if the liquid lens is placed near theCCD plane. The actual position of the rear focal plane of theobjective varies with the objective construction, but is gener-ally situated somewhere inside the objective barrel, except insome cases with objectives of low magnification. It is difficultto insert the liquid lens into the barrel of the objective, since theexternal diameter of the liquid lens is considerably bigger thanthe internal diameter of the barrel and electrical wiring of theliquid lens does not pass through the barrel. An optical relaysetup (Lee et al., 2013) can be used to obtain a correspondingplane to insert the liquid lens. But it is little complicated. Forthe convenience of the system setup and maintenance, the

liquid lens is chosen to place close to the rear plane of the ob-jective in this paper. The magnification constancy is sacrificedto some extent but within a permissible range. According toEquation (3), the magnification of the optical group with theconjugate object plane β1 increases linearly when the liquidlens diopter increases. It is worth noting that the value of β1

is negative, so that the absolute value of β1 decreases linearlywith an increased liquid lens diopter.

The distance between the conjugate object plane of the im-age plane and the object principle of the optical group is rep-resented as −l , according the Gauss’s formula in geometricaloptics, −l can be solved by the following equation:

1l ′ − 1

l= 1

f ′ . (5)

Since −l1 = −l − l H , then the distance between the conju-gate object plane of the image plane and the objective can bededuced as follows:

− l1 = f ′1

(d + l ′

2 − d l ′2φ2

)d − f ′

1 + l ′2 − d l ′

2φ2 + f ′1l ′

2φ2. (6)

Figure 2 shows the ray tracing of an object point in a fixedplane across the optical group as the liquid lens diopter is var-ied. Since the aperture of the optical group coincides with theliquid lens, the emergent chief ray does not change directionwhen the liquid lens diopter is changed, and the centre of thegrain formed on the CCD plane does not change either, whichresults in constant magnification with a fixed plane β2.

According to geometrical optics, the following equation canbe deduced:

β2 = l ′2

d(

l1

(1d + 1

f ′1

)− 1

) . (7)



25 30 35 40 45 50 554000

5000

6000

7000

8000

9000

10000

11000

12000

13000

voltage (V)

dept

h

depth vs. voltageCurve−fit result

Fig. 3. A curve fit for the relationship between the focus depth and the liquid lens driving voltage.

Letting r be the radius of the grain formed on the imageplane as the diopter of liquid lens is changed, we get:

r =∣∣∣∣∣Dl ′

2φ2 − D − D(

f1l ′2 + l1l ′

2

)d f1 + d l1 − f1l1

∣∣∣∣∣ , (8)

which is used in further point spread function (PSF) deduc-tion in section ‘EDOF and 3DI for the flexible-axial-capabilitymicroscope’.

Flexibly controlling the working distance of a microscope

The working distance of a commonplace microscope is fixed,so the samples need to be translated to the working plane of amicroscope to obtain a clear image. AF is generally employedby automatically moving the sample to the working planeof the microscope with the help of motors and translationstages. However, the motors and the translation stages areoften cumbersome, which is not desirable for the compactnessof the microscope system. Moreover, the error of the motionsystem restricts the focus precision of a microscope. To addressthis issue, the inserted liquid lens is used as a focusing elementto adjust the working distance. The small size of the liquidlens complements the compactness of the microscope and thevari-focus error of the liquid lens is small, which helps with AFprecision.

In this paper, we employ passive autofocusing based onimage-sharpness evaluation. The Tenengrad (Sun et al., 2004)operator is used in focus criterion and mount-climbing algo-rithm (He et al., 2003) is used here for finding positions ofbest focus. Here, the AF process consists of three steps: (1)determining the direction of the diopter change of the liquidlens in order to improve the focus evaluation of images; (2)coarse focusing in which the diopter steps of the liquid lens

Table 1. Curve-fit results for the relationship between the focus depth andthe liquid lens driving voltage.

Fitting function y = a + bc+U

Fitting result y = 2.745 × 104 + 9.735×105

−93.46+UR2 0.9999

vary as much as 1 m−1; (3) fine focusing in which the dioptersteps of the liquid lens vary as much as 0.1 m−1, which is theresolution of the liquid lens.

In order to find the relationship between the focus depth,the liquid lens driving voltage, and the impact on the magni-fication of the conjugate object plane of the CCD when usingthe liquid lens as a focus element, a calibration experimentwas conducted. The sample to be focused is placed on amotorized translation stage under the microscope objectiveand the distance between the objective and the sample isvaried by driving the translation stage. With every changeof depth, autofocusing with the liquid lens is performed toproduce a clear image of the sample. The depth of the sampleand the liquid lens driving voltage is then recorded, providinga set of focus-depth data relating to the liquid lens drivingvoltage. Since the liquid lens diopter is linear with the drivingvoltage according the Equation (6), the relationship betweenthe focus depth and the driving voltage of liquid lens can besimplified with the following equation:

y = a + bc + U

, (9)

where y represents the focus depth, e.g. the working distanceof our flexible-axial-capability of microscope, U represents theliquid lens driving voltage, and a , b, c are fit parameters.


2 1 6 Y . Q U A N D H . Y A N G

Table 2. Fit results for the magnification of the conjugate object planeand the liquid lens driving voltage.

Fitting function y = kU + mFitting result y = −0.1998U + 18.79R2 0.9999

The fitted results of this relationship are shown in Figure 3and Table 1, and the R2 value is 0.9999. According to theseresults, the working distance of the microscope can be adjustedfrom 4.5 to 12.5 mm by varying the liquid lens diopter. Also,the focus-depth range is 8 mm.

The relationship between the magnification of the conjugateobject plane of the CCD and the liquid lens driving voltage isalso calibrated, and the line-fit results are shown in Figure 4and Table 2. According to the line-fit results, samples placed atany depth within the working distance range can be measuredwith high precision.

EDOF and 3DI for the flexible-axial-capability microscope

By flexibly changing the working distance of a microscopeusing a liquid lens with the AF method, our flexible-axial-capability microscope can automatically acquire clear imagesof planar samples at an arbitrary depth within the focus-depthrange of the liquid lens. However, when imaging samples withdepth ranges exceeding the depth of field of the microscope,EDOF and 3DI methods need to be employed.

Two EDOF strategies are employed in the flexible-axial-capability microscope. We first discuss the volumetric opti-cal sampling method (Liu & Hua, 2011), which varies thefocus of the liquid lens with exposure time to acquire imageswith an increased depth range, though these are blurry dueto defocusing caused by focus sweeping of the liquid lens, butcan be restored by an image-restoration algorithm. The othermethod involves capturing a focal-stack sequence by changingthe liquid lens diopter and using the image-fusion algorithmto acquire EDOF images. The 3DI of samples is accomplishedby applying the shape-from-focus (Nayar, 1992) algorithm toa focal stack.

EDOF based on the volumetric optical sampling method

Volumetric optical sampling consists of two steps: (1) The fo-cus of the microscope is rapidly scanned through the depthrange of the sample by varying the focus of the liquid lens inan image integration period, therefore the captured image isan integration of both an in-focus and out-of-focus projectionof the 3D object on the 2D image plane and is a blurry inter-mediate image. (2) Postprocessing is then applied to restorethe all-in-focus EDOF image, utilizing the PSF of the opticalsystem.

In order to restore a clear image over the EDOF, the PSFhas to be determined. Assuming the PSF of the microscopewith the diopter fixed at φ through an image-acquisitionintegration time of PSF(φ), the PSF of the microscope usingvolumetric optical sampling while fast-scanning the liquidlens diopter from φ0 to φ1 with an averaged speed, is IPSF

25 30 35 40 45 50 558.5

9

9.5

10

10.5

11

11.5

12

12.5

13

13.5

voltage (V)

mag

nific

atio

n

magnification vs. voltageLine−fit result

Fig. 4. A line fit for the magnification of the conjugate object plane and the liquid lens driving voltage.



(integral point spread function). This can be calculated withfollowing equation, using a numerical integration method:

IPSF = 1φ1 − φ0

∫ φ1

φ=φ0

dφ · PSF(φ), (10)

where PSF(φ) can be represented as a function of the radiusof the grain from a point light source formed on the imageplane, which was deduced earlier in Equation (8). PSF(φ) canbe modelled as a pillbox function (Nagahara et al., 2008) if nooptical aberration exists:

P SF = 4πb2

�( r

b

), (11)

where � (x) is a rectangle function with a value of 1 if |x| <

1/2, and a value of 0 otherwise, and b is a constant. In thepresence of optical aberrations, PSF(φ) can be modelled as

PSF = 2

π (gb)2 exp(

− 2r 2

(gb)2

), (12)

where g and b are both constants.Since IPSF is invariant to scene depth (Nagahara et al.,

2009), the all-in-focus EDOF can be restored by deconvolv-ing the blurry intermediate image with a single IPSF. Manyalgorithms have been proposed for deconvolution (Gonzalez& Woods, 2007) such as Wiener filtering, Lucy–Richardsonfiltering and constrained least-square filtering (CLSF). In thispaper, constrained least-square filtering is employed becauseof its superior restoration performance and noise immunity.The solution for constrained least-square filtering in the fre-quency domain is

F̂ (u, v) =[

H ∗(u, v)

|H (u, v)|2 + γ |P (u, v)|2

]G (u, v), (13)

where γ is a Lagrangian multiplier, and can be selected bypreference, H (u, v) is the discrete Fourier transform (DFT) ofIPSF, G (u, v) is the DFT of the captured blurry image and

P (u, v) is the DFT of P (x, y) =⎡⎣ 0 −1 0

−1 4 −10 −1 0

⎤⎦ .

EDOF based on focal stack fusion

EDOF based on focal stack fusion was also realized in this study.The focal stack of the sample across its depth range is acquiredby capturing images with a varying liquid lens diopter, whichis altered in smallest-possible steps. Many methods for imagefusion (Aslantas & Toprak, 2014) have been proposed. In thispaper, Laplacian pyramid fusion (Aiazzi et al., 1999) has beenapplied to acquire all-in-focus images, because of its high speedand high fusion performance at multi-resolution levels. At first,

the Laplacian pyramid decomposition of images at k fusionlevels to is applied as follows:⎧⎨

⎩L i = G i − U P (G i+1) ⊗ g5×5 (1 ≤ i ≤ k − 1)

L k = G k . (14)

where UP (G i+1) indicates a projecting pixel at (x, y) in imageG i+1 to the position (2x + 1, 2y + 1) of a new image, ⊗ isthe symbol for the convolution operation, g5×5 is a Gaussiankernel with 5 × 5 pixel size and G 1 is the original image forthe Laplacian pyramid decomposition.

Then, fusion rules are applied on every level of the Lapla-cian pyramids of images. The method of weighted averagingis applied on the top level, since the data in the top-level imageconsist of basic colour and texture information. The methodof using the pixel with maximum absolute value is appliedin other levels of the pyramids, since they have a higher fre-quency of containing critical detail information. Finally, thefused image can be built from the Laplacian pyramid recon-struction as follows:{

G k = L k

G i = L i + UP (G i+1) ⊗ g5×5 (1 ≤ i ≤ k − 1), (15)

here G 1 is the result of fusion reconstruction.

3D reconstruction from the SFF algorithm

The 3D modelling of samples can be reconstructed from a focalstack using the SFF algorithm. At first, the focal stacks needto be registered in order to eliminate the image differencescaused by inconsistent magnification. Then, the depth map ofthe sample is calculated by selecting the depth of the pixels withthe maximum focus evaluation value across the focal stack.The sum-modified-Laplacian (SML) operator can be used to

Fig. 5. Gaussian interpolation of the focus evaluation curve near the peak.


2 1 8 Y . Q U A N D H . Y A N G

Fig. 6. Experimental setup: (A) schematic, (B) photograph.

calculate the focus evaluation value as follows:

F (i, j ) =i+N∑

x=i−N

j+N∑y= j−N

|2 f (x, y) − f (x − s, y) − f (x + s, y)|

+|2 f (x, y) − f (x, y − s) − f (x, y + s)|, (16)

where f (x, y) is the pixel value of the image at position (x, y),s is a variable that can be selected to accommodate for differenttexture patterns and N is the window size. In order to producea smooth surface in the 3D reconstruction, Gaussian interpo-lation is employed to calculate a better estimation of the depth.Figure 5 shows a typical focus evaluation curve of pixels at afixed position across the focal stack, which was established ear-lier. The points (dm−1, Fm−1), (dm, Fm), and (dm+1, Fm+1) arenear the depth with maximum focus value, and the estimateddepth is calculated using the following equation:

d = (ln Fm − ln Fm+1)(d 2

m − d 2m−1

) + (ln Fm − ln Fm+1)(d 2

m+1 − d 2m

)2 (ln Fm − ln Fm−1) (dm − dm−1) + 2 (ln Fm − ln Fm+1) (dm+1 − dm)

. (17)

Using the all-in-focus image acquired from the image-fusionalgorithm described above as the texture image, the 3D modelof the sample can be acquired.

Experimental setup and results

In order to verify that the presented flexible-axial-capabilitymicroscope is able to flexibly change the working distance,extend the depth of field, and render a 3D model of observedsample comprehensively, an off-the-shelf microscope was con-structed, with a liquid lens inserted below the objective. Fig-ure 6 shows the experimental setup of the microscope. Thisflexible-axial-capability microscope consists of a measurementmicroscope (Nikon, Japan) with an 10× /0.25 objective, aCCD detector (Basler, Germany), an image receiver with an

IEEE-1394 interface (Point Gray Research, Canada), a liquidlens and controller (Varioptic, France), and a host computer(3.10 GHz × 2 CPU, 4.0 GB RAM). The diopter range for theliquid lens is (–5 m−1, +15 m−1). The liquid lens is affixed tothe rear plane of the objective with connecting pieces, shownin Figure 7. The depth of field of the microscope without theliquid lens was computed to be 8.5 μm. After inserting theliquid lens, the depth of field of the microscope, without usingEDOF, extends to 20 μm by measurement. This is caused byrestrictions on the aperture of the microscope system with thesmall-aperture liquid lens.

Flexibly varying the microscope working distance

According to the calibration results from Figure 3, the rangeof the working distance of the microscope is 4.5–12.5 mm.

In order to verify the variable working distance function ofthe microscope as well as the magnification fitting result, amicrocalibration board is placed 5, 8 and 12 mm under themicroscope, and AF by varying the liquid lens diopter is utilizedto acquire clear images. Images with consistent magnificationare obtained by zooming in or out with the magnification fac-tor calibrated in advance. Figure 8 shows images before liquidlens AF, after liquid lens AF, and after the registration of sam-ples at three different depths. Results show that the workingdistance of the microscope can be controlled by liquid lensAF, and consistent magnification can be obtained by zoom-ing the image in or out according to the calibration factor.Table 3 shows the compared results for the speed and error ofsix AF tests, including liquid lens AF and AF via a motorizedtranslation stage. Results show that the liquid lens AF is faster



Fig. 7. Mechanical connecting pieces designed to affix the liquid lens below the objective. (A) The upper part, which is connected to the revolvingnosepiece. (B) The lower part, which is connected to the objective lens. (C) The combination of the parts with liquid lens set inside.

Fig. 8. AF results for a microcalibration board at three different depths. (A) Image before liquid lens AF at 5 mm. (B) Image after liquid lens AF at 5 mm.(C) Image after registration at 5 mm. (D) Image before liquid lens AF at 8 mm. (E) Image after liquid lens AF at 8 mm. (F) Image after registration at 8mm. (G) Image before liquid lens AF at 12 mm. (H) Image after liquid lens AF at 12 mm. (I) Image after registration at 12 mm.

and has a smaller focus error compared with translation-stageAF.

EDOF using the volumetric optical sampling method

Using the volumetric optical sampling method described in sec-tion ‘EDOF based on the volumetric optical sampling method’,the depth of field of the microscope is extended to 200 μm fromthe variable working plane of the microscope. Figure 9 showsEDOF results of a microsensor. The depth between samplesin the green and red rectangles is 160 μm by measurement.Figure 9 (A) shows the image of the microsensor capturedunder a fixed diopter of liquid lens. The image in the greenrectangle is in focus and the image in the red rectangle isblurry. Figure 9 (B) shows the blurry image of the microsen-sor captured using volumetric optical sampling method. The

Table 3. Comparison of AF by liquid lens and AF by stage translation.

Depth of AF by liquid lens AF by stage translation

No. sample Time (ms) Error (V) Time (ms) Error (μm)

1 5 mm 389 0 625 152 481 0 700 103 8 mm 156 0 389 144 200 0 456 65 12 mm 356 0 756 156 423 0 826 9Average

value– 334.2 0 625.3 11.5


2 2 0 Y . Q U A N D H . Y A N G

Fig. 9. Comparison of images of a microsensor captured with a fixed liquid lens focus and using the volumetric optical sampling method. (A) Image of themicrosensor captured with a fixed liquid lens diopter. (B) Blurry intermediate image of the microsensor captured using the volumetric optical samplingmethod. (C) Clear image of the microsensor restored from (B).

Fig. 10. Focus stacks of a soldier joint captured by varying the liquid lens focus, with EDOF and 3DI. (A) The image of the soldier joint when varying theliquid lens to focus the microscope at 8.5 mm. (B) The image of the solder joint when varying the liquid lens to focus the microscope at 8.8 mm. (C) EDOFimage of the solder joint acquired from fusing a focal stack. (D) A three-dimensional reconstruction result of the solder joint from the focal stack.

image in the green rectangle and the red rectangle are bothblurred to the same extent. Figure 9 (C) shows the clear imageof microsensor, which is restored from (B) using CLSF algo-rithm; the image in the green and red rectangles are bothclear. This provides verification of the EDOF of our microscopeusing methods of volumetric sampling.

EDOF and 3DI based on a focal stack

Using the methods for acquiring a focal stack and algorithmsfor image fusion and 3D reconstruction, tests of EDOF and3DI methods were conducted and the results are shown inFigure 10. Thirty images of a soldier joint on a circuit boardare captured as the liquid lens diopter changes its resolutionby 0.1 m−1, and two images of the focal stack are shown in

Figures 10 (A) and (B). The EDOF image, which is acquired bythe Laplacian pyramid fusion algorithm, is shown in Figure 10(C). Figure 10 (D) shows the 3D reconstruction result from theSFF algorithm.

Comprehensive functions for flexible-axial-capability microscopy

In this section, experiments are described for determining com-prehensive functions for flexible-axial-capability microscopy,including flexibly changing the working distance, EDOF and3DI. At first, flexibly changing the working distance of the mi-croscope is applied to sample focusing, then EDOF or 3DI areapplied to acquire information over a long distance.

Figure 11 shows observation results for strands of the Se-taria grass with flexible-axial-capability microscopy at varying



Fig. 11. Images of strands of the Setaria grass. (A) Image of the strand when the flexible-axial-capability microscope is focused at 8.9 mm. (B) The blurryintermediate image is produced using volumetric optical sampling methods within a depth range of 400 μm at a focal distance of 8.9 mm. (C) Imagerestoration result of (B). (D) Image of the strand when the flexible-axial-capability microscope is focused at 9.5 mm (E) The blurry intermediate imageproduced with volumetric optical sampling methods within the depth range of 400 μm at a focal distance of 9.5 mm. (F) Image restoration result of (E).

depths from the objective. Figures 11 (A)–(C) show imagesof a strand at a distance of 8.9 mm from the microscope.Figure 11(A) is an image of the strand when the microscopeis focused at a depth of 8.9 mm made by varying the focus ofthe liquid lens. Since the depth range of the strand at 8.9 mmexceeds the depth of field of the microscope, the image is partlyin focus (in red rectangle) and partly obscured (in green rect-angle). In order to get an all-in-focus image of the strand, Thedepth of field is extended to 400 μm from the working planeof the microscope by the volumetric optical sampling method.Figure 11(B) is the blurry intermediate image of the strand us-ing the volumetric optical sampling method, and Figure 11(C)

is the image restoration result of Figure 11(B) where it canbe seen that the image part in red rectangle and green rect-angle are both clear. Figures 11(D)–(F) are the correspondingimages of the strand at 9.5 mm from the microscope. FromFigures 11(A) and (D), the process of flexibly changing theworking distance of the flexible-axial-capability microscope isverified. From Figures 11(A) and (C) and Figures 11(D) and(F), the process of EDOF around an arbitrary working plane ofthe flexible-axial-capability microscope is verified.

Figure 12 shows EDOF and 3DI results of the letter Ton a coin placed at different depths from the microscopeusing focal-stack methods. The coin is placed at 9.7 and

Fig. 12. Images of the letter T on a coin at different depths using focal-stack methods. (A) Image of the letter placed at 9.7 mm below the objective,without EDOF. (B) Image-fusion result from a focal stack of the coin placed at 9.7 mm. (C) Three-dimensional reconstruction result of the coin placedat 9.7 mm. (D) Image of the letter placed at 11.5 mm, without EDOF. (E) Image-fusion result from a focal stack with the coin placed at 11.5 mm. (F)Three-dimensional reconstruction result from a focal stack of the coin placed at 11.5 mm.


2 2 2 Y . Q U A N D H . Y A N G

11.5 mm successively under the microscopy, and a focalstack is acquired by varying the focus of the liquid lens. TheEDOF images and 3D models of the sample are producedusing the Laplacian pyramid fusion and shape-from-focusalgorithms.

Conclusions

In this paper, a flexible-axial-capability microscope is designedby inserting a liquid lens into an off-the-shelf microscope andutilizing methods of volumetric optical sampling, image fu-sion, and 3D reconstruction from a focal stack. Theoreticalanalysis on the working distance and the magnification ofthe microscope when varying the focus of liquid lens was dis-cussed. The liquid lens is positioned at the rear plane of theobjective in order to produce a relatively small change in mag-nification. Autofocusing using the liquid lens is employed toflexibly control the focal distance of the microscope and makeit convenient for observing samples placed at different depths.Volumetric optical sampling methods, image-fusion and 3D re-construction from a focal stack are employed to acquire EDOFand 3DI. Experiments validated that the proposed system canobserve and measure a sample from 4.5 to 12.5 mm and thatthe depth of field can be extended to 400 μm, which is 20 timesof the depth of field of the basis microscope. Therefore, withtwo simple mechanical connecting pieces, combined with al-gorithms for AF, EDOF and 3DI, a liquid lens can be insertedinto an off-the-shelf microscope to perform long distance, largedepth-of-field observation and measurement.

Acknowledgements

This work is supported by the National Natural Science Foun-dation of China under Grant No. 51105027.

References

Aiazzi, B., Alparone, L., Barducci, A., Baronti, S. & Pippi, I. (1999) Multi-spectral fusion of multisensor image data by the generalized Laplacianpyramid. In IEEE International Geoscience and Remote Sensing Symposium2. pp. 1183. Hamburg, Germany.

Arai, J., Yamashita, T., Miura, M., Hiura, H., Okaichi, N., Okano, F. &Funatsu, R. (2013) Integral three-dimensional image capture equip-ment with closely positioned lens array and image sensor. Opt. Lett. 38,2044–2046.

Aslantas, V. & Toprak, A.N. (2014) A pixel based multi-focus image fusionmethod. Opt. Commun. 332, 350–358.

Conchello, J.-A. & Lichtman, J.W. (2005) Optical sectioning microscopy.Nat. Methods 2, 920–931.

Gonzalez, R.C. & Woods, R.E. (2007) Digital Image Processing. 3rd edn.Prentice Hall, Upper Saddle River, New Jersey.

He, J., Zhou, R. & Hong, Z. (2003) Modified fast climbing search auto-focus algorithm with adaptive step size searching technique for digitalcamera. IEEE Transac. Consum. Electron. 49, 257–262.

Kemper, B., Langehanenberg, P. & Von Bally, G. (2007) Digital holo-graphic microscopy. Optik. Photonik. 2, 41–44.

Lee, M., Gibson, G., Bowman, R., Bernet, S., Ritsch-Marte, M., Phillips,D. & Padgett, M. (2013). A multi-modal stereo microscope based on aspatial light modulator. Opt. Express 21(14), 16541–16551.

Levoy, M., Ng, R., Adams, A., Footer, M. & Horowitz, M. (2006) Lightfield microscopy. ACM Transac. on Graphics (Proc. SIGGRAPH) 25(3),924–934.

Liang, Q. & Qu, Y. (2012) A texture–analysis–based design method forself-adaptive focus criterion function. J. Microsc. 246, 190–201.

Liu, S. & Hua, H. (2011) Extended depth-of-field microscopic imaging witha variable focus microscope objective. Opt. Express 19, 353–362.

Nagahara, H., Kuthirummal, S., Zhou, C. & Nayar, S.K. (2008) Flexibledepth of field photography. In Proceedings of the European Conference onComputer Vision 4. pp. 60–73. Marseille, France.

Nayar, S.K. (1992) Shape from focus system. In Proceedings of the ComputerVision and Pattern Recognition. pp. 302–308. Champaign, Illinois.

Qu, Y., Liu, L. & Yang, H. (2012) Extended depth-of-field for visual systems:an overview. In Proceedings of the SPIE, 8558, 88580K, Beijing, China.

Schulein, R., DaneshPanah, M. & Javidi, B. (2009) 3D imaging with axiallydistributed sensing. Opt. Lett. 34, 2012–2014.

Sun, Y., Duthaler, S. & Nelson, B.J. (2004) Autofocusing in computermicroscopy: selecting the optimal focus algorithm. Microsc. Res. Tech.65, 139–149.

Sun, Y., Duthaler, S. & Nelson, B.J. (2005) Autofocusing algorithm selec-tion in computer microscopy. In Proceedings of the IEEE /RSJ Interna-tional Conference on Intelligent Robots and Systems. pp. 70–76. Edmonton,AB, Canada.

Wu, Z., Wang, D. & Zhou, F. (2012) Bilateral prediction and intersectioncalculation autofocus method for automated microscopy. J. Microsc.248, 271–280.


optical microscopy with flexible axial capabilities using a...

Documents