stereo-mate generation of high-resolution satellite imagery using a parallel projection model

11
Stereo-mate generation of high-resolution satellite imagery using a parallel projection model Howook Chang, Kiyun Yu, Hyunseung Joo, Yongil Kim, Hyejin Kim, Jaewan Choi, Dong Yeob Han, and Yang Dam Eo Abstract. Synthesis methods to create a stereo-mate of satellite imagery from an orthophoto have been developed in many previous studies. If these methods are applied in an urban area where there are many adjacent tall buildings, stereo viewing is inhibited by occlusion in the orthophoto and its stereo-mate. In high-resolution satellite imagery, the in-track view angle of the image is usually far from vertical; consequently, the occluded area near tall structures occupies a large area, and this severely affects stereo viewing. This study proposes a different approach to creating stereo-mates for high-resolution satellite imagery by projection of the digital surface model (DSM) draped by the original single image onto a fictitious satellite sensor model. The main benefit of this method is enhanced stereo viewing by arranging the fictitious sensor model to reduce occlusion area. The physical sensor model of the original image is previously derived by parallel projection model, and then the stereo-mate fictitious sensor model is determined from the physical sensor model. Résumé. Des méthodes de synthèse conçues pour créer des stéréomates d’images satellitaires à partir d’une orthophoto ont déjà été développées dans de nombreuses études. Si l’on applique ces méthodes dans un milieu urbain, où y il a plusieurs bâtiments contigus en hauteur, la vision stéréo est limitée par une occlusion dans l’orthophoto et son stéréomate. En imagerie satellitaire haute résolution, l’angle de visée dans la trace de l’image est généralement éloigné de la verticale si bien que la région occultée près des structures élevées occupe une grande surface affectant ainsi sévèrement la vision stéréo. La présente étude propose une approche différente pour la création de stéréomates d’images satellitaires haute résolution en projetant le modèle numérique de surface (MNS) drapé de l’image originale unique sur un modèle fictif de capteur satellitaire. Le principal avantage de cette méthode est qu’elle améliore la vision stéréo en ajustant le modèle fictif du capteur pour permettre la réduction de la zone d’occlusion. Le modèle physique du capteur de l’image originale est dérivé au départ par le biais d’un modèle de projection parallèle et le modèle fictif de capteur du stéréomate est déterminé par la suite à partir de ce dernier. [Traduit par la Rédaction] 67 Introduction The demand for stereo viewing using high-resolution satellite images is increasing in many applications, such as archaeological exploration (Jauregui et al., 2004), fault-zone detection (Salvi, 1995), landscape architecture, and tourist map generation, because stereo viewing enables an enhanced observation of a wide area. Notwithstanding its great requirement, the acquisition of a high-resolution stereo pair is still costly and time consuming. Moreover, the acquired stereo pair is likely to have incorrect characteristics for stereo viewing (Chang et al., 2006). It usually has nonzero y-parallax and an inappropriate base to height ratio. Synthesis of a pair from a single image is therefore a promising alternative. Producing a stereo pair from a single satellite image implies the generation of a stereo-mate by combining it with a digital surface model (DSM). Generally, a stereo-mate is synthesized from an orthophoto (Salvini et al., 2004; Wang, 2004). In this method, each pixel is shifted in proportion to its z value in the digital terrain model (DTM). Because this method was intended to work with much smoother topographic relief, it could not be used with a detailed DSM and a true orthoimage. When synthesizing a stereo-mate using a DSM and an orthoimage, it will have been created with a different viewing angle than those of the original image acquisition, leading to the problem that informationless occluded parts in the orthoimage will become visible on the stereo-mate, creating voids therein. Because the occluded area is not assigned a brightness value and is left empty, it inhibits a comfortable perception of depth, and therefore distorts stereo viewing. With high-resolution satellite imagery, the in-track view angle of the satellite image is usually far from vertical, such that the occluded area occupies a large proportion of the stereo pair. Thus, it leaves tall © 2008 CASI 57 Can. J. Remote Sensing, Vol. 34, No. 2, pp. 57–67, 2008 Received 29 November 2007. Accepted 3 April 2008. Published on the Canadian Journal of Remote Sensing Web site at http://pubs.nrc-cnrc.gc.ca/cjrs on 18 July 2008. H. Chang, K. Yu, 1 H. Joo, Y. Kim, H. Kim, and J. Choi. School of Civil, Urban & Geo-System Engineering, Seoul National University, Sinlim-Dong, Kwanak-Gu, Seoul 151-742, Korea. D.Y. Han. Department of Civil & Enviromnmental Engineering, Chonnam National University, San96-1, Dunduk-Dong, Yeosu, Jeonnam 550-749, Korea. Y.D. Eo. Department of Advanced Technology Fusion, Konkuk University, 1 Hwayang-Dong, Gwangjin-Gu, Seoul 143-701, Korea. 1 Corresponding author (e-mail: [email protected]).

Upload: yang-dam

Post on 13-Apr-2017

212 views

Category:

Documents


0 download

TRANSCRIPT

Stereo-mate generation of high-resolutionsatellite imagery using a parallel projection model

Howook Chang, Kiyun Yu, Hyunseung Joo, Yongil Kim, Hyejin Kim, Jaewan Choi,Dong Yeob Han, and Yang Dam Eo

Abstract. Synthesis methods to create a stereo-mate of satellite imagery from an orthophoto have been developed in manyprevious studies. If these methods are applied in an urban area where there are many adjacent tall buildings, stereo viewingis inhibited by occlusion in the orthophoto and its stereo-mate. In high-resolution satellite imagery, the in-track view angleof the image is usually far from vertical; consequently, the occluded area near tall structures occupies a large area, and thisseverely affects stereo viewing. This study proposes a different approach to creating stereo-mates for high-resolutionsatellite imagery by projection of the digital surface model (DSM) draped by the original single image onto a fictitioussatellite sensor model. The main benefit of this method is enhanced stereo viewing by arranging the fictitious sensor modelto reduce occlusion area. The physical sensor model of the original image is previously derived by parallel projectionmodel, and then the stereo-mate fictitious sensor model is determined from the physical sensor model.

Résumé. Des méthodes de synthèse conçues pour créer des stéréomates d’images satellitaires à partir d’une orthophoto ontdéjà été développées dans de nombreuses études. Si l’on applique ces méthodes dans un milieu urbain, où y il a plusieursbâtiments contigus en hauteur, la vision stéréo est limitée par une occlusion dans l’orthophoto et son stéréomate. En imageriesatellitaire haute résolution, l’angle de visée dans la trace de l’image est généralement éloigné de la verticale si bien que larégion occultée près des structures élevées occupe une grande surface affectant ainsi sévèrement la vision stéréo. La présenteétude propose une approche différente pour la création de stéréomates d’images satellitaires haute résolution en projetant lemodèle numérique de surface (MNS) drapé de l’image originale unique sur un modèle fictif de capteur satellitaire. Le principalavantage de cette méthode est qu’elle améliore la vision stéréo en ajustant le modèle fictif du capteur pour permettre laréduction de la zone d’occlusion. Le modèle physique du capteur de l’image originale est dérivé au départ par le biais d’unmodèle de projection parallèle et le modèle fictif de capteur du stéréomate est déterminé par la suite à partir de ce dernier.[Traduit par la Rédaction]

67Introduction

The demand for stereo viewing using high-resolutionsatellite images is increasing in many applications, such asarchaeological exploration (Jauregui et al., 2004), fault-zonedetection (Salvi, 1995), landscape architecture, and tourist mapgeneration, because stereo viewing enables an enhancedobservation of a wide area. Notwithstanding its greatrequirement, the acquisition of a high-resolution stereo pair isstill costly and time consuming. Moreover, the acquired stereopair is likely to have incorrect characteristics for stereo viewing(Chang et al., 2006). It usually has nonzero y-parallax and aninappropriate base to height ratio. Synthesis of a pair from asingle image is therefore a promising alternative. Producing astereo pair from a single satellite image implies the generationof a stereo-mate by combining it with a digital surface model(DSM).

Generally, a stereo-mate is synthesized from an orthophoto(Salvini et al., 2004; Wang, 2004). In this method, each pixel isshifted in proportion to its z value in the digital terrain model(DTM). Because this method was intended to work with muchsmoother topographic relief, it could not be used with a detailedDSM and a true orthoimage. When synthesizing a stereo-mateusing a DSM and an orthoimage, it will have been created witha different viewing angle than those of the original imageacquisition, leading to the problem that informationlessoccluded parts in the orthoimage will become visible on thestereo-mate, creating voids therein.

Because the occluded area is not assigned a brightness valueand is left empty, it inhibits a comfortable perception of depth,and therefore distorts stereo viewing. With high-resolutionsatellite imagery, the in-track view angle of the satellite imageis usually far from vertical, such that the occluded areaoccupies a large proportion of the stereo pair. Thus, it leaves tall

© 2008 CASI 57

Can. J. Remote Sensing, Vol. 34, No. 2, pp. 57–67, 2008

Received 29 November 2007. Accepted 3 April 2008. Published on the Canadian Journal of Remote Sensing Web site athttp://pubs.nrc-cnrc.gc.ca/cjrs on 18 July 2008.

H. Chang, K. Yu,1 H. Joo, Y. Kim, H. Kim, and J. Choi. School of Civil, Urban & Geo-System Engineering, Seoul NationalUniversity, Sinlim-Dong, Kwanak-Gu, Seoul 151-742, Korea.

D.Y. Han. Department of Civil & Enviromnmental Engineering, Chonnam National University, San96-1, Dunduk-Dong, Yeosu, Jeonnam550-749, Korea.

Y.D. Eo. Department of Advanced Technology Fusion, Konkuk University, 1 Hwayang-Dong, Gwangjin-Gu, Seoul 143-701, Korea.

1Corresponding author (e-mail: [email protected]).

building sidewalls as empty in the stereo-mate of theorthophoto. This problem originating from the fact that avertical sidewall has few DSM points in a raster structuredisturbs stereo viewing. To solve such problems, a differentmethod is required to generate high-resolution stereo-mates.

This paper proposes a method that produces the stereo-mateof a satellite image by the projection of the DSM draped by theoriginal image onto a fictitious satellite sensor model to reducethe occluded area in a stereo pair. It is first necessary to extractthe original image physical sensor model to define the stereo-mate fictitious sensor model. Usually, only the rationalpolynomial coefficient (RPC) is provided instead of thephysical sensor model for high-resolution satellite imagery, andthe parallel projection model is adopted to extract the originalimage physical sensor model from its RPC (Morgan, 2004).The original image physical sensor model is used to constrainthe stereo-mate fictitious sensor model. This means that thestereo-mate sensor model is determined with the constraint thatthe original image physical sensor model imposes to reduce theoccluded area and provide zero y-parallax. DSM usuallyregisters few points in vertical sidewalls, particularly those ofbuildings, as mentioned previously. Therefore, such sidewallsare left empty in a stereo-mate derived from the projection ofDSM points. This paper resolves this problem by addingartificial DSM points along vertical sidewalls at regularintervals. This process is described in the Experiments andresults section. Projection of the artificially added points ontothe stereo-mate sensor model fills the empty vertical sidewallareas with their own brightness values. Because artificial DSMpoints are also draped by the original image, they assign abrightness value to the empty stereo-mate vertical wall areas.

The proposed stereo-mate generation approach has thefollowing advantages. First, the occlusion that disturbs stereoviewing is reduced considerably when compared to that of astereo pair produced from orthophotos. Second, the absence ofbrightness value in tall building sidewalls is restored by addingartificial DSM points. This reduction of unfilled areas such asocclusions and empty sidewalls gives better stereo viewing.Third, the synthesized stereo-mate can be used for spaceintersection because it also has a physical sensor model.

Related workStereo-mate generation from aerial photographs has been

developed in previous studies since its introduction in 1968 byCollins (1969). Such research into stereo-mate generation fromaerial photographs is not discussed in this paper becausesatellite imagery has characteristics that differ from those ofaerial photography. In contrast with aerial photographs, high-resolution satellite images are usually acquired by a pushbroomsensor with an in-track view angle far from vertical. Stereo-mate generation for high-resolution satellite imagery must beapproached with a different point of view.

Batson et al. (1976) proposed a method for generating astereo-mate by introducing artificial parallax in a geometricallycorrected Landsat image. Although this approach was

appropriate for low- and medium-resolution satellite images, itis doubtful whether it could be applied with a high-resolutionimage. The occlusion caused by tall buildings that did notmanifest itself in low- and medium-resolution imagery was notconsidered in this research. O’Neill and Dowman (1988)proposed the generation of a stereo-mate by projecting DSMpoints onto fictitious sensor models, and there are somesimilarities to his method in this study. However, his researchdid not impose any constraint in determining the stereo-matefictitious sensor model, and therefore it did not deal withproblems such as occlusion and nonzero y-parallax. In addition,it used ephemeris information that is not usually available withhigh-resolution satellite imagery, which generally providesonly RPC. Moreover, its iterative calculation of the ray tracingemployed to register DSM with the original SPOT imageimpedes computation speed and capacity.

Salvi (1995) and Usery (1993) synthesized stereo-mates ofLandsat and SPOT imagery, respectively. Because theysynthesized stereo-mates from orthophotos, their approach isnot appropriate to high-resolution satellite imagery of urbanareas, as mentioned. Salvini et al. (2004) synthesized a stereo-mate of a high-resolution QuickBird image. However, he onlyconsidered geological landforms and also synthesized a stereo-mate from orthophotos. Therefore, it is questionable if hisresearch is appropriately applied to urban areas where there aremany adjacent tall buildings.

Objective and workflowThe objective of this paper is to set out a method to produce a

stereo mate of a high-resolution satellite image for qualitystereo viewing. To achieve this, a stereo-mate is created basedon the original single image parallel projection sensor model.The significance of this method is tested by occlusion areameasurement and 3D coordinate accuracy. Figure 1 shows theflow chart of the proposed method.

First, a given RPC model is corrected by manually matchingthe single image and DSM points from which arbitrary groundcontrol points (GCPs) are generated using the corrected RPCs(Di and Li, 2003). Second, the original single image parallelprojection sensor model and its roll angle (ψ) are extracted bythe arbitrary GCPs. Third, a fictitious sensor model of thestereo-mate is determined with reference to the original imageparallel projection sensor model to reduce occlusion in thestereo-mate and provide zero y-parallax. Fourth, artificial DSMpoints are added into the vertical walls, and all DSM points areregistered with the original image according to the extractedsensor model. Lastly, the stereo-mate is synthesized byprojecting the DSM registered with the original image onto thestereo-mate sensor model.

In this paper, stereo-mates are produced from bothorthophotos and the proposed method to permit furthercomparison. The two stereo-mates are compared with respect toocclusion size and distribution. Three-dimensional coordinate

58 © 2008 CASI

Vol. 34, No. 2, April/avril 2008

accuracy testing and epipolarity analysis are also performed toillustrate the validity of the proposed method.

The rest of this paper is organized as follows: parallelprojection sensor model extraction is described; the stereo-mate fictitious sensor model determination is explained; theexperimental process and results are discussed; the significanceof this proposed method is tested; and the conclusion is given.

Pushbroom imaging geometry andperspective to parallel (PTP) transform

When RPC is provided without any ephemeris data, directlyextracting a satellite image physical (rigorous) sensor model isimpossible. Therefore, an indirect method of parallel projectionmodelling is herein employed to extract the physical sensormodel. A satellite image is transformed to a parallel projectionimage, and then the physical sensor model is extracted in theform of a parallel projection sensor model. Prior to this,understanding of the imaging geometry of high-resolutionsatellite imagery is necessary. Although a satellite image isobtained by a pushbroom scanner, the image approximates thattaken by parallel projection when the apparent field of view(AFOV) of the image is comparatively narrow (Okamoto et al.,1992; Ono et al., 1999). To apply the parallel projection modelto the satellite pushbroom imaging geometry, threeassumptions are necessary:

(1) The satellite image should have a very narrow AFOV.The AFOV of high-resolution satellite imagery is inreality less than 1° because of its very high altitude.

(2) Satellite attitude should be constant during the imagescanning. Because the image is achieved within a veryshort time, this assumption is also reasonable. Thisassumption is necessary to approximate that the bundlesdefined by consecutive scans are parallel.

(3) The satellite velocity should be constant during imagecapture. This assumption is also realistic for the samereason as set out before and is necessary to simplify thegeometric relationship between consecutive scans.

If these assumptions are satisfied, application of the parallelprojection model is permissible along the satellite flighttrajectory, but not along the across-track direction as shown inFigure 2, which shows that the satellite linear pushbroomimaging geometry approximates to parallel projection onlyalong the flight trajectory. Along the across-track direction,perspective projection is shown to be dominant.

To employ the parallel projection model to extract the originalimage physical sensor model, the perspective projection alongthe across-track direction should be transformed to parallelprojection. Perspective to parallel (PTP) transform is used toconvert perspective projection to parallel projection (Okamoto etal., 1992). PTP transform assumes a flat terrain, and the error

© 2008 CASI 59

Canadian Journal of Remote Sensing / Journal canadien de télédétection

Figure 1. Workflow of the proposed method.

resulting from this assumption is mentioned in the Conclusion.Figure 3 illustrates PTP transform.

In Figure 3, point O is the perspective centre of thepushbroom scanner, and the satellite flight trajectory isperpendicular to the plane, where c is the principal distance,and ψ is the roll angle. The parallel projection direction is set tobe the optical axis direction. If the image plane is translated toplane E, the object space point P on the flat terrain is mapped to

P1 through perspective projection and P2 through parallelprojection. Basically, PTP transformation is the process thattransforms P1 to P2 before applying scale down to the originalimage plane. The transformation is finally expressed by thefollowing equation (Morgan, 2004):

y yy

c

P2 P1P1

=−

1

1 tan ψ(1)

where yP1 is the coordinate of the perspective projection alongthe scanner direction, yP2 is the coordinate of the parallelprojection along the scanner direction, c is the principaldistance, and ψ is the roll angle. The tangent of ψ in the triangleAP2P and the similarity between triangle PP2P1 and OAP1 areused for the derivation. By Equation (1), every pixel of thesatellite image is transformed along the across-track directionafter ψ is extracted using arbitrary GCPs.

Extraction of parallel projection sensormodel

After the transformation of the original pushbroom image tothat of parallel projection by PTP, the parallel projection modelis applied to the transformed image. Figure 4 shows thegeometry of parallel projection.

60 © 2008 CASI

Vol. 34, No. 2, April/avril 2008

Figure 2. Linear pushbroom imaging geometry.

Figure 3. PTP transform.

An object space point P on the ground is mapped to theimage plane by the parallel ray. The constant vector (L, M, N)stands for the parallel ray direction. Assuming (L, M, N) as aunit vector, it can be represented by only two parameters,namely L and M. Additional to the vector (L, M, N), theparameters for the orientation and location of the image planeshould be determined. The orientation of the image plane isexpressed by ω, φ, and κ. The location of the image plane originrelative to the ground coordinate origin O is expressed by ∆x,∆y, and ∆z; ∆z is set to zero for the sake of convenience becausethe same scene will be generated regardless of the ∆z value. Theimage plane is located where it includes the ground coordinateorigin O because ∆z is zero. Lastly, the scale s is determined toscale down the spatial extent of the projected scene. Insummary, the eight parameters L, M, ω, φ, κ, ∆x, ∆y, and s shouldbe determined to describe parallel projection geometry. Whenthe point P(X, Y, Z) is mapped to p(u, v, 0) in the image, therelation between (X, Y, Z) and (u, v, 0) is described as follows:

u

v

X

Y

Z

L

M

N0

=

+

R R( , , ) ( , ,ω φ κ ω φ κ)λ

(2)

where λ is the distance between the object space point P and thescene point p, and R(ω, φ, κ) is the rotation matrix between theobject and scene coordinate system. Then, the image is scaleddown by the scale parameter s and relocated to an x and ycoordinates system by the translation of ∆x and ∆y. Thisprocess is shown in the following:

x

y s

X

Y

Z

s

L

M

N0

=

+R R( , , ) ( , ,ω φ κ ω φ κ)λ

+

∆∆

x

y

0

(3)

where ∆z is zero.Equation (3) can be rearranged by eliminating λ using the

third row of the matrix. It is reduced to linear equations asfollows:

x A X A Y A Z A

y A X A Y A Z A

= + + += + + +

1 2 3 4

5 6 7 8

(4)

The eight parameters A1–A8 represent the eight parameters L,M, ω, φ, κ, ∆x, ∆y, and s, respectively. In summary, the relationbetween the 3D object space and the image coordinate systemcan be described with two linear equations in the parallelprojection model.

Because the original satellite image is a pushbroom image,applying Equation (4) directly to it is impossible. Equation (4)can be applied to the image only after it is converted by a PTPtransformation as presented in Equation (1). Consequently,combining Equations (1) and (4) is necessary to utilize theparallel projection model. The term yP2 in Equation (1)corresponds to y in Equation (4) because both represent theparallel projection coordinate along the across-track direction.They are therefore used as a parameter to combine Equations(1) and (4). As a result, the following equation is derived:

x A X A Y A Z A

yA X A Y A Z A

cA X A Y

= + + +

=+ + +

+ + +

1 2 3 4

15 6 7 8

5 61P tan

A Z A7 8+ )

(5)

where x is the coordinate along the flight trajectory, and yP1 isthe coordinate along the scanner direction before PTPtransform.

The nine parameters in Equation (5), namely A1–A8 and ψ,can be extracted using the minimum five X–Y–Z GCPs. In thispaper, nine parameters for the original image are obtainedusing arbitrary GCPs from the RPC model first, and then thephysical sensor model (L, M, ω, φ, κ, ∆x, ∆y, and s) is extractedfrom A1–A8 by transformation between Equations (3) and (4).The transformation between Equations (3) and (4) is describedin detail in Morgan (2004). In the next section, the stereo-matesensor model is determined based on the original imagephysical sensor model (L, M, ω, φ, κ, ∆x, ∆y, and s).

© 2008 CASI 61

Canadian Journal of Remote Sensing / Journal canadien de télédétection

Figure 4. Geometry of parallel projection.

Determination of the stereo-mate sensormodel

After the original satellite image parallel projectionparameters L, M, ω, φ, κ, ∆x, ∆y, and s are extracted, the stereo-mate fictitious sensor model is determined based on them.Before determining this, the original image is converted by PTPtransformation of Equation (1) using the ψ obtained fromEquation (5). It is then projected to the horizontal plane. Thereason the original image is thus projected is that the x-parallaxof the synthetic stereo pair becomes proportional to DSMheight in the parallel projection model only when the syntheticpair is generated on the horizontal plane. The parallelprojection parameters ωh and φh should each be zero to representthe horizontal plane, κh and sh are set to be the same as theoriginal image k and s, and ∆xh and ∆yh are selected to limit thespatial extent of the projected image within positivecoordinates.

After transforming the original image, the parallel raydirection of the stereo-mate Ls, Ms, and Ns is determined. Thefirst condition to be considered is that the intersection anglebetween the parallel ray vectors of the stereo-mate and theoriginal image should be set between 2° and 5°, as shown inFigure 5.

If the intersection angle is too large, two problems occur.One is that large occlusion areas appear in the stereo-mate, andthe other is that a given object takes a totally different shape inthe original image and the stereo-mate. These problemsseverely disturb stereo viewing. If the intersection angle is toosmall, the artificial x-parallax will be insufficient to perceivedepth adequately. If the intersection angle is determinedbetween 2° and 5°, the following equation is formulated:

( , , )( , , )L M N L M N LL MM NNs s s s s s= + +

= + + + +L M N L M N2 2 2 2 2 2s s s cos θ

= cos θ (�parallel ray vectors are unit vectors) (6)

where (L, M, N) is the parallel ray vector of the original image,(Ls, Ms, Ns) is the parallel ray vector of the stereo-mate, and θ isthe intersection angle between (L, M, N) and (Ls, Ms, Ns).

The next condition to be considered to determine the stereo-mate parallel ray vector is that the y-parallax should beeliminated because this disturbs the stereo viewing. Equation(7) should be satisfied to eliminate the y-parallax. The outerproduct of (L, M, N) and (Ls, Ms, Ns) is normal to the epipolarplane, and it should be normal to the x axis to eliminate y-parallax:

R L M N L M N( , , ){( , , ) ( , , )} ( , , )ω φ κh h h s s s× =� 1 0 0 0 (7)

where ωh, φh, and κh are orientation parameters of the horizontalplane; and (1, 0, 0) stands for the x axis. The vector (Ls, Ms, Ns)that simultaneously satisfies Equations (6) and (7) becomes thestereo-mate parallel ray vector. Because (Ls, Ms, Ns) is a unitvector, two equations are sufficient to determine (Ls, Ms, Ns).Besides (Ls, Ms, Ns), the remaining stereo-mate parallelprojection parameters (ωs, φs, κs, ∆xs, ∆ys, and ss) are set to be thesame as the horizontal plane parameters (ωh, φh, κh, ∆xh, ∆yh, andsh) to remove y-parallax.

Experiments and resultsThe stereo-mate generation process is performed according

to the following steps: (1) arbitrary GCPs are extracted from theoriginal image RPC; (2) the original image roll angle ψ andparallel projection sensor model are extracted using arbitraryGCPs and Equations (3)–(5); (3) the original image goesthrough PTP transformation using Equation (1) and is projectedto the horizontal plane; (4) the stereo-mate parallel ray vector isdetermined using Equations (6) and (7); (5) the rest of thestereo-mate parallel projection parameters are set to be thesame as the horizontal plane parameters; (6) artificial DSMpoints are added to vertical walls; (7) DSM points are registeredwith the original image; and (8) DSM points registered with theoriginal image are projected onto the stereo-mate sensor modeland, as a result, the stereo-mate is completed. Figure 6 brieflyillustrates the whole process.

Addition of the artificial DSM points is described here.These are added because they prevent the vertical wall area inthe stereo-mate from being left empty, and they remove double-mapping in registering DSM with the original image. DSMslopes above a certain threshold can here be consideredvertical. In our case, when the threshold is 12, a vertical wall isassumed to exist, and artificial DSM points with an interval of0.5 m are added to this wall as shown in Figure 7. Theseartificial points are located at the same X–Y position as thehigher DSM point, and their z values are assigned accordingly.

The stereo-mate of a QuickBird satellite image was producedthrough the combination of DSM and the original image. Thestereo-mates produced from orthophotos were compared withthose produced using the proposed method. The result of thiscomparison is presented and discussed in this section.

62 © 2008 CASI

Vol. 34, No. 2, April/avril 2008

Figure 5. Determination of the stereo-mate ray direction.

© 2008 CASI 63

Canadian Journal of Remote Sensing / Journal canadien de télédétection

Figure 6. Stereo-mate generation process (steps 1–7).

Figure 7. Artificial DSM points addition.

Experimental data

The QuickBird image used in this experiment coveredDaejeon City in South Korea. The geographical coordinates ofthe area cover 36.328° to 36.382° latitude north and 127.336° to127.403° longitude east. The type of image was QuickBirdStandard Ortho-ready, which had been processed only byradiometric correction and had a ground sampling distance ofabout 0.55 m. The surface point coordinates of DSM used inthis experiment were produced by light detection and ranging(lidar) data with 0.5 m resolution.

Implementation and results

The first stereo pair was synthesized from the QuickBirdimage’s orthophoto, and the second stereo pair was synthesizedby the method proposed in this study. The two stereo pairs werecompared visually and quantitatively. Figure 8 shows thestereo pair synthesized from the orthophoto.

Because large occlusion areas occurred around tall buildingsin both the orthophoto and its stereo-mate, stereo viewing wasseverely disturbed. The discomfort in stereo viewing is obviouswhen the stereo pair is viewed through an anaglyph, as shownin Figure 9a. The reason for the large occlusion in the stereopair synthesized from the orthophoto is quite simple. Althoughthe in-track view angle of high-resolution satellite imagery isgenerally far from vertical, the ray bundles of the imagery aredeformed to vertical when the orthophoto is generated. As aresult of this deformation, a large occlusion area is generated inthe orthophoto stereo pair.

In contrast with the stereo pair synthesized from theorthophoto, the stereo-pair synthesized by the proposed methodhad much less occlusion. This is shown in Figure 10, where theimage in Figure 10a is the original converted by PTPtransformation and projected to the horizontal plane, and the

image in Figure 10b is the stereo-mate synthesized by theproposed method.

Contrary to the stereo-mate synthesis based on theorthophoto, the proposed method preserved the original imageray bundle and set the stereo-mate ray vector as similar to thatof the original image. As a result, the ray bundle deformationwas minimized and occlusion was reduced considerably.Moreover, the occlusion area was scattered like salt-and-peppernoise as seen in Figure 10, whereas large solid occlusion areasoccurred in the orthophoto stereo pair, as seen in Figure 8.Restoration of massive occlusion by interpolation was almostimpossible. On the other hand, filling in noise-like occlusion bytriangulated irregular network (TIN) interpolation wasrelatively easy and efficient. TIN interpolation of the noise-likeocclusion efficiently eliminated the empty areas that haddisturbed stereo viewing.

The anaglyph in Figure 9b was produced from the stereopair synthesized by the proposed method and restored by TINinterpolation. It delivered better perception of depth than theexample in Figure 10a synthesized from the orthophotobecause its occlusion was restored efficiently. The quantitativecomparison of the two stereo pairs also revealed that theoccluded area was reduced when using the proposed method. Inthe stereo pair synthesized from the orthophoto, the occludedproportion occupied 21.5% of the total area. When the stereopair was synthesized from the proposed method, the occludedproportion was only 5.5% of the total area (Table 1). Thisquantitative result was obtained when the standard deviation ofthe DSM z value in the object area was about 23 m and theangular distance between the original image ray bundledirection and the vertical was about 30°.

Although the numerical quantity may vary according to theobject area building density and the original satellite image raybundle direction, much reduction of occlusion area in the

64 © 2008 CASI

Vol. 34, No. 2, April/avril 2008

Figure 8. Stereo pair synthesized from the orthophoto: (a) orthophoto, (b) stereo-mate.

synthetic stereo usually happens when the proposed method isapplied to high-resolution satellite images where there aremany adjacent tall buildings in an urban area. In summary, theoccluded area in the synthetic stereo was reduced when theproposed method was used, and reduced occlusion enhancedthe perception of depth for stereo viewing, as shown inFigure 9. The anaglyph in Figure 9b produced by the proposed

method enabled better 3D viewing than that produced from theorthophoto (Figure 9a).

Accuracy analysisThe significance of the proposed method is tested in this

section. It is considered with respect to its 3D coordinateposition accuracy and y-parallax. The stereo-mate produced inthis paper had parallel projection parameters such as Ls, Ms, Ns,ωs, φs, κs, ∆xs, ∆ys, and ss. Therefore, it was possible to test the3D coordinate position accuracy of the stereo-mate through thespace intersection with the original image transformed by PTPtransformation. As a result of space intersection using 20 realGCPs, a position error of 1.93 m was observed. This is assumedto have occurred for three reasons. First, the DSM generated

© 2008 CASI 65

Canadian Journal of Remote Sensing / Journal canadien de télédétection

Figure 10. Stereo pair synthesized by the proposed method: (a) original; (b) stereo-mate.

Figure 9. (a) Anaglyph produced from Figure 8 (orthophoto-based methodology). (b) Anaglyph produced fromFigure 10 (proposed method). Red and cyan glasses are necessary for stereo viewing.

Occluded area /total area (%)

Stereo pair synthesized from the orthophoto 21.5Stereo pair synthesized by the proposed method 5.5

Table 1. Comparison of occluded area in the stereo pairs.

from lidar data originally included 1.145 m of position error ascompared with that from real GCPs. Second, arbitrary GCPsthat were generated to extract the original image parallelprojection sensor model also contained a small position errorbecause even the corrected RPCs were not perfect. Third, theflat terrain assumption for PTP transformation introduced someposition error. The error derived by assuming flat terrain can beestimated using the following equation (Morgan, 2004):

∆ ∆y s Z=+

sin( )

cos( )

αψ α

(8)

where ∆y is the error introduced by the flat terrain assumption,∆Z is the height deviation from the average elevation, and α ishalf of the field angle of the scanner.

In the QuickBird image used in the experiment, ψ is less than1 and s (scale) is 0.257 × 10–5. Assuming that α is less than 1, asis the case in high-resolution satellite imagery, ∆y is calculatedto be within 13.75 µm (1 pixel) when ∆Z is less than 30 m. Thisresult indicates that the error introduced by a flat terrainassumption does not lead to significant distortion in stereoviewing when the height deviation from the mean elevation isless than 30 m. Actually, the object area had considerable relief,and the standard deviation of the DSM z value was about 23 m.Table 2 shows the result of the 3D position accuracy test. Inaddition to the 3D position accuracy, the stereo pair was testedfor y-parallax removal. Equation (7) gives the necessarycondition to eliminate y-parallax. To test whether y-parallaxwas eliminated, the scale invariant feature transform (SIFT)(Lowe, 2004) was used to extract corresponding points betweenthe original image projected on the horizontal plane and thesynthesized stereo-mate. Eighty-nine correspondences wereextracted, and the root mean square y-parallax determined was0.335 pixel. In other words, y-parallax existed within a halfpixel in the synthesized stereo pair. A 3D position error ofabout 1.9 m and y-parallax within a half pixel were satisfactoryfor providing quality stereo viewing. According to Scharstein(1996), a relatively small error in canonical interpretation of 3Dgeometry is acceptable in stereo viewing. Because thegeometric error occurring in the synthesized stereo pair wasvery small relative to the scale of the object area and the image,it was concluded that the synthesized stereo pair acquiredenough accuracy to provide quality stereo viewing.

ConclusionThe original satellite image physical sensor model was

extracted in this paper using the parallel projection model, and

a stereo-mate was synthesized based on this model. The stereo-mate sensor model was determined to reduce occlusion andeliminate y-parallax. The main interest within the proposedmethod was that the original ray bundle was preserved and thatit acted as a constraint in selection of the stereo-mate sensormodel. As a result, the information contained in the originalimage was fully preserved, and less occlusion in the syntheticstereo was achieved. The addition of artificial digital surfacemodel (DSM) points within vertical walls also eliminatedunfilled empty space in the stereo-mate wall areas.

Because the synthesized stereo-mate had a parallelprojection sensor model, its significance was proved throughspace intersection. In addition, y-parallax of the syntheticstereo was within a half pixel. Even though a small error in 3Dposition and y-parallax existed in the synthetic stereo, the errorswere negligible in stereo viewing. In conclusion, the methodproposed in this paper enabled better stereo viewing of high-resolution satellite imagery by reducing occlusion and filling inthe empty vertical wall, whereas synthetic stereo derived fromorthophotography did not permit successful stereo viewing.

AcknowledgementsThe authors gratefully acknowledge financial support from

Seoul Research and Business Development (10540) and fromthe SNU SIR BK21 research program funded by the Ministry ofEducation & Human Resources Development.

ReferencesBatson, R.M., Edwards, K., and Eliason, E.M. 1976. Synthetic stereo and

Landsat pictures. Photogrammetric Engineering & Remote Sensing,Vol. 42, No. 10, pp. 1279–1284.

Chang, H.W., Kim, H.J., Lee, J.B., and Yu, K.Y. 2006. Synthesis of stereo-mate through the fusion of a single aerial photo and LiDAR data. InProceedings of the ASPRS International Symposium on Remote Sensing, 2–4 November 2006, Busan, Korea. CD-ROM.

Collins, S.H. 1969. The accuracy of optically projected orthophotos andstereo-orthophotos. The Canadian Surveyor, Vol. 24, No. 5, pp. 450–463.

Di, K., and Li, R.X. 2003. Rational functions and potential for rigorous sensormodel recovery. Photogrammetric Engineering & Remote Sensing, Vol. 69,No. 1, pp. 33–41.

Jauregui, M., Jauregui, L., Chacon, L., and Vilchez, J. 2004. Digital stereo-orthophotos of archaeological sites from small format aerial photographs.In Proceedings of the 20th ISPRS Congress, Commission 5, 12–23 July2004, Istanbul, Turkey. pp. 461–466.

Lowe, D.G. 2004. Distinctive image features from scale-invariant keypoints.International Journal of Computer Vision, Vol. 60, No. 2, pp. 91–110.

Morgan, M. 2004. Epipolar resampling of linear array scanner scenes. Ph.D.dissertation, Department of Geomatics Engineering, University of Calgary,Calgary, Alta.

Okamoto, A., Akamatsu, S., and Hasegawa, H. 1992. Orientation theory forsatellite CCD line-scanner imagery of hilly terrains. In Proceedings of the17th ISPRS Congress, Commission 2, 2–14 August 1992, Washington, D.C.

66 © 2008 CASI

Vol. 34, No. 2, April/avril 2008

No. ofGCPs

Root mean square error (RMSE; m) Total position error(displacementaverage; m)X direction Y direction Z direction

20 1.543 1.386 0.664 1.93

Table 2. Three-dimensional (3D) coordinate accuracy test.

International Archives of Photogrammetry and Remote Sensing, Vol. 29,pp. 217–222.

O’Neill, M.A., and Dowman, I.J. 1988. The generation of epipolar syntheticstereo-mates for SPOT images using DEM. In Proceedings of the 16thISPRS Congress, Commission 2, Kyoto, Japan. pp. 587–598.

Ono, T., Honmachi, Y., and Ku, S. 1999. Epipolar resampling of highresolution satellite imagery. In Proceedings of the Joint Workshop of ISPRSWG I/1, I/3, and IV/4: Sensors and Mapping from Space, 27–30 September1999, Hannover, Germany.

Salvi, S. 1995. Analysis and interpretation of Landsat synthetic stereo pair forthe detection of active fault zones in the Abruzzi region. Remote Sensing ofEnvironment, Vol. 53, pp. 153–163.

Salvini, R., Anselmi, M., Rindinella, A., and Callegari, I. 2004. QuickBirdstereo photogrammetry for geological mapping. In Proceedings of the 20thISPRS Congress, Commission 7, 12–23 July 2004, Istanbul, Turkey.pp. 1101–1104.

Scharstein, D. 1996. Stereo vision for view synthesis. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition (CVPR96),18–20 June 1996, San Francisco, Calif. IEEE, New York. pp. 852–858.

Usery, E.L. 1993. Virtual stereo display techniques for three-dimensionalgeographic data. Photogrammetric Engineering & Remote Sensing,Vol. 59, No. 12, pp. 1737–1744.

Wang, M. 2004. A new approach for generating a measurable seamless stereomodel based on mosaic ortho-image and stereo-mate. In Proceedings of the20th ISPRS Congress, Commission 3, 12–23 July 2004, Istanbul, Turkey.pp. 122–127.

© 2008 CASI 67

Canadian Journal of Remote Sensing / Journal canadien de télédétection