Real Time Superresolution by Means of anUltrasonic Light Diffractor and TV System

Takuso Sato, Mitsuhiro Ueda, and Tamon Ikeda

The .change of angle and the shift of frequency of light by an ultrasonic light diffractor are used for realtime realization of a holographic superresolution system. A TV system and an electrical filter are usedto pick up the desired image hologram from the superposed, images, which are obtained by a number ofobject-beam lights and reference beams. For two-dimensional as well as one-dimensional objects, imagessuperresolved three to five times are displayed on a TV monitor in real time.

IntroductionSuperresolution systems have been discussed com-

prehensively by Lukosz' and several methods of real-ization have been proposed.2 -5 We also proposed amethod that utilizes the holographic technique. 6' 7

In our method, the object is illuminated coherentlyfrom different angles and the images are superposedsequentially as image holograms with correspondingangles of reference light and the superresolved imageis obtained from the resulting multiexposed holo-gram. But it takes quite a long time to obtain thesuperresolved image because of the need to changethe angle of object and reference beams mechanical-ly, photographic multiexposure of holograms, andprocessing.

In this paper a method that overcomes these dif-ficulties is proposed. In this method, the setting ofthe angles of object and reference beams is done byusing the diffraction of light by ultrasonic waves.The effect of frequency shift of diffracted light is.used for the simultaneous superposition of desiredholograms. That is, when a number of object andreference beams fall on a hologram, the frequencyshift of the diffracted light is used so as to form thestationary fringes between pairs of object and refer-ence beams that contribute to the formation of thesuperresolved image and the traveling fringes be-tween the other pairs of object and reference beams.Consequently the image hologram, which is obtainedby the object and reference beams from the ultrason-ic light diffractor, is scanned by a video camera andthe desired information that corresponds to the su-

The authors are with the Research Laboratory of Precision Ma-chinery and Electronics, Tokyo Institute of Technology, Ohokaya-ma, Meguro-ku, Tokyo 152, Japan.

Received 25 June 1973.

perposed image hologram is picked up by passing thevideo signal through a high pass filter and envelopedetector and the resulting superresolved image isdisplayed on a TV monitor in real time.

PrincipleAt first let us review the method of superresolution

by holography. In the system of Fig. 1(a), if theobject is illuminated from the direction 0, only theinformation that corresponds to low spatial frequen-cy components of the object is passed through thelimiting aperture of the optical system. But if theillumination from the direction of 0-1 is used, the in-formation that corresponds to the high spatial fre-quency component of the object can be transmitted.Therefore by illuminating the object sequentiallyfrom the directions o, 01, 0 and at the same timeby changing the directions of the reference beam 00,O:, 0-1, respectively, we can obtain a multiexposedimage hologram whose carrier frequency changes inaccordance with the images that are obtained underdifferent angles of object illumination. As a result ofdifferent carrier frequencies, the images composed ofhigh frequency components of the object leave thehologram at different angles as compared to theangle of the image of the low frequency components.Since this situation corresponds to image formationusing a lens of larger aperture as shown in Fig. 1(b),a three-times superresolved image is reconstructedfrom the hologram.

For the system shown in Fig. 1 the conditions re-quired to extend the spatial bandwidth withoutoverlap and gap in the spatial frequency range are asfollows.

The intensities of the object beam I,, must be ofthe same value for all n and the intensities of the ref-erence beam Rn must be of the same value for all n:

1,, = Cl (n = O +1, ... , N),R., = C2 (n =0, +1, ... , N),

(1)

(2)

1318 APPLIED OPTICS / Vol. 13, No. 6 / June 1974

E 01 d ' d2

2a hologram

Obet (a) RI

6a

- K l. RiX

object RL hologram

( b)Fig. 1. Explanation of superresolution by holography. (a) Sup-erposition of image holograms. (b) Reconstruction of superre-solved image. OI, object illumination; RL, reference light; RI, re-

constructed image.

where C1 and C2 are constants and these conditionsare necessary to obtain flat characteristics of a re-sponse function of the superresolved system.

If we take kO as the offset angle of the referencebeam, the relations between angles of object beam fOn

and angles of reference beam On must be as follows:

sin,, = M(sink, - sino,), (3)

sin0,, = 2an/d, (n = 0 i1, . .. , N), (4)

where M = d2/dL is the lateral magnification of theoptical system, a is the radius of the lens aperture,and the maximum number of N is limited by thecondition sinON < 1.0. Conditions (3) and (4) arenecessary to extend the spatial bandwidth withoutoverlap and gap in order to obtain a faithful recon-struction of the object.7 Then the spatial frequencybandwidth becomes (2N + 1) times as large as theoriginal bandwidth of the system.7

Now let us introduce the ultrasonic light diffractorfor setting the angles of the object and referencebeams and the simultaneous superposition of the ho-lograms. In the ultrasonic light diffractor, which isshown schematically in Fig. 2(a), the following rela-tions are well known. The intensity I,', the frequen-cy f,,, and the diffraction angle 0fo' of the diffractedlight of the order n are represented in terms of thefrequency Jo of the incident light, the wavelength Xof the light, the wavelength A of the ultrasonic wave,the frequency fi of the ultrasonic wave, and theRaman-Nath parameter v of the diffractor as follows:

fn = fo + n f,

sin0', = n-(A/A),I = J 2(U) (n = 0, +1, A2, . . .),

Now by comparing the relations (4) and (6), weobtain

2a/d, = X/A. (8)

If this is satisfied, the conditions required for the an-gles of the object beam are satisfied simultaneouslyfor all values of n. As for the reference waves, in thecase where the angle 0 is small, the relation

(1/M)0n = ( - 0) (n = 0, 1 .. , N) (9)

should be satisfied. But this relation is satisfied au-tomatically if M = 1 and the diffracted lights areused directly as reference beams. When M # 1, theangles of the reference beam can be adjusted bypassing the diffracted light through a magnifying op-tical system or by using proper optical arrangements.As the directions are known beforehand, this ar-rangement does not seem to be very difficult.

To satisfy the relations (1) and (2) the value of theRaman-Nath parameter v should be chosen properly.For example, if v = 1.43 is chosen, the condition Io'= I,,' is satisfied and in this case the conditions (1)and (2) are satisfied for n = O. i1 only. On theother hand, if v = 3 is chosen, we can see that inten-sities Io', I1', I-2' are of about the same magnitude.Thus, as for the value of n = 1 or 2 we may choose vso that Eqs. (1) and (2) are roughly satisfied. More-over, if the cascade modulators is used, the control ofthe intensity of a given order of diffracted light canbe done more easily.

Now, as the same light from the diffractor is usedfor the object and reference beams, the image holo-grams are obtained on the image plane of the sys-tem. Moreover, according to the relation (5) theimage obtained by a given diffraction order of theobject beam makes stationary fringes only with thesame order of reference beam. Other cross termsbetween different orders of object and referencebeams do not make stationary holograms; they maketraveling holograms. Thus, only the desired holo-grams make stationary fringes. In this way the re-quired superposed image hologram is obtained as astationary hologram and the superposition is estab-lished at once.

The mean frequency of the fringes of these holo-grams is determined by the offset angle 0O of the ref-erence light. Thus, if a video camera is placed atthe image plane and if the image is scanned acrossthe fringe pattern, the information of the stationary

(5)

(6)

(7)

where Jn is the Bessel function of the order n and theRaman-Nath parameter v indicates the phase excur-'sion of the light at the outlet of the ultrasonic lightdiffractor. A linear relation exists between theRaman-Nath parameter and the voltage applied tothe ultrasonic transducer of the ultrasonic light dif-fractor. 9' 10 Although relation (7) is valid only forthe low ultrasonic frequency (typically 0.5-10 MHz),experiments were done in this region.

IL

I_1

10

1.1,

( a ) ( b)Fig. 2. Ultrasonic light diffractor. (a) One-dimensional. (b)Two-dimensional. T, ultrasonic transducer; WT, water tank;

SA, sound absorber; IL, incident light; DL, diffracted light.

June 1974 / Vol. 13, No. 6 / APPLIED OPTICS 1319

Fig. 3. Real-time superresolution system for unity magnifica-tion. ULD, ultrasonic light diffractor; AM, amplifier; SG, sinus-oidal signal generator; M, mirror; HM, beam splitter; L, lens (f= 400 mm); L2, lens (f = 200 mm); OB, object plane; CA, aper-ture 2a = 0.5 mm; VC, video camera; BPF, bandpass filter; ED,

envelope detector; MO, TV monitor; 1 = 12 = 400 mm.

K \( \ a.

* 0 Orderx 1st Order

A -1st Order

o 2nd Order

m -2nd Order

amount of the multiple of the ultrasonic frequency.Consequently the carrier frequency of the holo-grams may change according to the difference of theoptical frequencies. But as it is a very small fractionof the frequency of light, as for the optical systemused in our experiment the change of the carrier fre-quency caused by the difference of the opticalfrequencies may be considered negligible.

To apply this principle for two-dimensional sys-tems, only a two-dimensional ultrasonic light diffrac-tor is required, which is shown in Fig. 2(b). Toavoid the stationary cross terms between diffractedlight in two different directions the two frequenciesof the ultrasonic waves must be shifted from eachother. The amount of the frequency shift is deter-mined by the temporal characteristics of the videocamera. Since it scans the images at the rate of 50frames/sec, the frequency shift between two ultra-sonic waves must be larger than 50 Hz.

Experimental Results

Real time superresolution experiments were carredout by using the system shown in Fig. 3. Since thesystem was used at unit magnification (M = 1), thediffracted light was used directly as the referencelight. In Fig. 4 the characteristics of the ultrasoniclight diffractor are shown as the relation between theamplitude of the applied voltage and the relative in-tensity of diffracted light. It may be seen from thisfigure that Io' = I1±' is satisfied at about 32 V. Thefrequency f of the ultrasonic waves was 3MHz anddistilled water was used as the medium of the dif-fractor. The cutoff frequency of the high pass filterwas set at 1 MHz from the relation of offset angle ofthe reference light and the scanning speed of thevideo camera. Other dimensions of the system wereas shown in Fig. 3 and they were adjusted so that therequired relation (8) was satisfied. In order to equa-lize the optical paths of object and reference beamsbetween the ultrasonic light diffractor and the video

00 10 20 30 40 50 60 70

V P-P(v)

Fig. 4. Characteristics of the ultrasonic diffractor. Vp-,, peak-to-peak voltage applied to the ultrasonic transducer; curves show

relative intensities of various diffraction orders.

holograms is transformed into a video signal. Bypicking up only this information using an electricalfilter, we can eliminate the noise that is due to thecross terms. In this way the superposed hologram isobtained as a video signal and after passing throughan envelope detector it is displayed on the TV moni-tor instantly.

In this method, as can be seen from the relation(5), the frequencies of the light of different illumi-nating angles are shifted from each other by the

(a) (b) (c) (d) (e)Fig. 5. Results of the superresolution for one-dimensional object.(a) Image of the object with a lens of 2a = 30 mm without ultra-sonic waves. (b) Image of the object with a lens of 2a = 1.5 mmwithout ultrasonic waves. (c) Image of the object with a lens of2a = 0.5 mm without ultrasonic waves. (d) Superresolved imagewhen 32 V of ultrasonic signal was applied (2a = 0.5 mm). (e)

Superresolved image when 75 V was applied (2a = 0.5 mm).

1320 APPLIED OPTICS / Vol. 13, No. 6 / June 1974

1.0

0.9

0.7

0.6

0.5

0.4

0.3

0.2

0.1

(a) (b) (c) (d)

Fig. 6. Results of the two-dimensional superresolution. (a)Image of the object with a lens of 2a = 30 mm without ultrasonicwaves. (b) Image of the object with a lens of 2a = 1.5 mm with-out ultrasonic waves. (c) Image of the object with a lens of 2a =0.5 mm without ultrasonic waves. (d) Superresolved image whenfi = 3.000 MHz, f2 = 3.002 MHz, and 32 V of ultrasonic signalswere applied at the same time to the ultrasonic transducers, re-spectively (2a = 0.5 mm). The object was so large that two TV

images were combined to give a full picture.

(a) (b) (c)

Fig. 7. Effects of band-pass filter and frequency shift. (a) Su-perresolved image with the frequency shift but without the highpass filter. (b) Superresolved image with both the filter and thefrequency shift. (c) Superresolved image with the filter but with-

out the frequency shift.

camera, the same imaging lens as used in the path ofthe object beam is inserted in the path of the refer-ence beam. Since the phase of the hologram signalis determined by the phase difference between theobject and reference beam, the phase differences inlights of various diffraction orders caused by differ-ent illuminating angles have no effect on the phasesof hologram signals. Consequently a flat phasecharacteristic of the response function of the superre-solved system is assured.

Experimental results for the one-dimensionalobject with different spatial frequencies are shown inFig. 5. Figure 5(a) corresponds to the original pat-tern, (c) is the image in the case where the radius ofthe aperture was limited to a = 0.25 mm, and (b) isthe image when the aperture of a = 0.75 mm wasused; it corresponds to three-times resolution.Then, by using the same system as in (c) we ap-plied the ultrasonic wave and tried the superresolu-tion. Figure 5(d) shows the superresolved image

when the applied voltage was 32 V. In this case,about three times superresolution was expected andby comparing the result with Fig. 5(b) we can clearlysee the expected result in Fig. 5(d). By increasingthe applied voltage up to 75 V we could see a furtherincrease of resolution as shown in Fig. 5(e). In thiscase, as can be seen from Fig. 4, the effect of second-order diffracted lights is involved and the resolutionincreases by a factor of 5.

In Fig. 6 the results for two-dimensional superreso-lution are shown. The two ultrasonic frequencieswere set at f, = 3.000 MHz and 2 = 3.002 MHz.From these results we can see the applicability ofthis method to two-dimensional systems.

In Fig. 7 the effects of the electrical filter and themutual shift of frequencies between two ultrasonicwaves for a two-dimensional superresolution systemare shown. The superresolved image is shown inFig. 7(b). When the filter is not used, the image be-comes as shown in Fig. 7(a) and the image cannot beseen because of the noise from cross terms. On theother hand, when the frequencies are adjusted to thesame value (f = 2 = 3.000 MHz), the cross termsbetween them make stationary fringes as shown inFig. 7(c) and the desired image becomes unresolved.'Experiments for a magnifying system in the case ofM = 2.0 were also carried out and results obtainedwere about the same.

Conclusions

The ultrasonic light diffractor and TV system withan electrical filter and an envelope detector are ap-plied to the real time realization of one-dimensionaland two-dimensional superresolution systems. Im-ages superresolved three to five times can be dis-played on the TV monitor in real time. This meth-od may also be considered as an extension of Lu-kosz's method,' which uses the two moving gratingsby the introduction of coherent background, but asthe offset angle and the TV system with the high-pass filter that picks up the desired signal play animportant part and as the system is explained moreclearly by using the holographic terms, the explana-tions in this paper are derived as an extension of theholographic superresolution system.7

References

1. W. Lukosz, J. Opt. Soc. Am. 57, 932 (1967).2. M. A. Grimm and A. W. Lohmann, J. Opt. Soc. Am. 56,1151

(1966).3. A. Bachl and W. Lukosz, J. Opt. Soc. Am. 57, 163 (1967).4. A. W. Lohmann and D. P. Paris, Appl. Opt. 3, 1037 (1965).5. A. I. Kartashev, Opt. Spectrosc. 9, 204 (1960).6. M. Ueda and T. Sato, J. Opt. Soc. Am. 61, 418 (1971).7. M. Ueda, T. Sato, and M. Kondo, Optica Acta 20, 403 (1973).8. T. Sato, K. Iwadate, G. Yamagishi, and M. Ueda, J. Opt.

Soc. Am. 61, 1716 (1971).9. M. Born and E Wolf, Principles of Optics (Pergamon Press,

New York, 1970), p. 593.10. R. Adler, IEEE Spectrum 4, 42 (May 1967).

June 1974 / Vol. 13, No. 6 / APPLIED OPTICS .1321