Comput Geosci (2011) 15:53–67DOI 10.1007/s10596-010-9195-1

ORIGINAL PAPER

Simulation of marine controlled source electromagneticmeasurements using a parallel fourier hp-finiteelement method

David Pardo · Myung Jin Nam · Carlos Torres-Verdín ·Michael G. Hoversten · Iñaki Garay

Received: 23 September 2009 / Accepted: 4 June 2010 / Published online: 9 July 2010© Springer Science+Business Media B.V. 2010

Abstract We introduce a new numerical method tosimulate geophysical marine controlled source electro-magnetic (CSEM) measurements for the case of 2Dstructures and finite 3D sources of electromagnetic(EM) excitation. The method of solution is based ona spatial discretization that combines a 1D Fouriertransform with a 2D self-adaptive, goal-oriented, hp-Finite element method. It enables fast and accurate

The work reported in this paper was funded by Chevron,the Spanish Ministry of Science and Innovation under theprojects TEC2007-65214, PTQ-08-03-08467, andMTM2010-16511.

D. Pardo (B) · I. GarayBasque Center for Applied Mathematics (BCAM),Bilbao, Spaine-mail: pardo@bcamath.org, dzubiaur@gmail.com

D. PardoIKERBASQUE, Basque Foundation for Science,Bilbao, Spain

M. J. NamDepartment of Energy and Mineral Resources Engineering,Sejong University, Sejong, Korea

C. Torres-VerdínThe University of Texas at Austin,Austin, Texas, USA

M. G. HoverstenChevron, San Ramon, California, USA

I. GarayInstitute for Theoretical Physics III,University of Erlangen-Nürnberg,Erlangen-Nuremberg, Germany

simulations for a variety of important, challenging andpractical cases of marine CSEM acquisition. Numericalresults confirm the high accuracy of the method aswell as some of the main physical properties of marineCSEM measurements such as high measurement sen-sitivity to oil-bearing layers in the subsurface. In ourmodel, numerical results indicate that measurementscould be affected by the finite oil-bearing layer by asmuch as 104% (relative difference). While the emphasisof this paper is on EM simulations, the method can beused to simulate different physical phenomena such asseismic measurements.

Keywords Marine controlled source electromagnetics(CSEM) · Fourier finite element method ·hp-adaptivity · Goal-oriented adaptivity

1 Introduction

1.1 History and main principles of marineCSEM measurements

Resistivity measurements have been used during thelast 80 years to quantify the spatial distribution of elec-trical conductivity in hydrocarbon-bearing reservoirs.Electrical conductivity is utilized to assess materialproperties of the subsurface, and is routinely used byoil-companies to estimate the volume of hydrocarbons(oil and gas) existing in a reservoir.

Borehole resistivity measurements have been rou-tinely acquired by oil-companies since 1927, when theSchlumberger brothers recorded resistivity data for thefirst time in Pechelbronn, France. However, marine

54 Comput Geosci (2011) 15:53–67

CSEM measurements have only been used by oil com-panies in the late 1990s and early 2000s, despite theearly academic research performed during the 1970sand 1980s on the topic (see [4, 6, 9]).

Marine CSEM measurements utilize a transmitterthat is typically located a few meters (10–100 m) abovethe sea-floor and is moved by a ship (see Fig. 1). Thistransmitter operates at frequencies in the range of 0.25–1.25Hz in order for EM waves to reach long distances.Measurements are recorded by a set of receivers lo-cated along the sea-floor at a distance of up to 20 kmfrom the source.

EM waves excited in marine CSEM travel throughdifferent materials in the path from the transmitterto the receivers. Since the media is lossy, EM waves

diffuse most of their energy before arriving to thereceivers. In addition, EM waves are also reflected atthe interfaces between different materials (see Fig. 1,bottom panel). The signal traveling through the sub-surface is the one that is useful to study the presence,location, and shape of hydrocarbon reservoirs. For thatpurpose, there is a need to remove the direct signalsthat travel trough the sea and the one reflected by thesea-air interface. Due to the low electrical conductivityof the air, the signal reflected by the air in a shallowwater environment may be several orders of magni-tude larger than the one traveling through the subsur-face, which increases the noise-to-signal ratio, therebyincreasing the difficulty of determining the presenceand location of hydrocarbons within the reservoir [33].

Fig. 1 Two views of a typicalmarine CSEM acquisitionsystem composed of onetransmitter carried by a ship,several receivers located atthe seafloor and severalsubsurface layers. EM wavesare reflected by differentmaterial interfaces with theair and the subsurface layers(see bottom panel)

Comput Geosci (2011) 15:53–67 55

For a more detailed physical interpretation of marineCSEM measurements, see [34].

1.2 Simulations of marine CSEM measurements

To improve the interpretation of results obtained withresistivity measurements, and thus, to better quantifyand determine existing subsurface materials and in-crease hydrocarbon recovery, diverse methods havebeen developed to perform numerical simulations (forinstance, [15, 17, 18, 22]) as well as to invert marineCSEM measurements [1, 5, 12, 16, 21].

Despite the existing numerical work recently devel-oped to simulate marine CSEM measurements, severalchallenges need to be overcome to produce moreaccurate simulations in a limited amount of time.Specifically, the main numerical challenges are the fol-lowing: (a) we need to consider in the same simulationobjects with very different geometries and sizes, includ-ing small antennas and large subsurface layers, (b) weneed to consider full EM effects, since direct current(DC) simulations at zero frequency do not providesufficient accuracy due to the large size of the com-putational domain, (c) we need to consider electricallyanisotropic formations, since this feature is essential toproperly characterize reservoirs, (d) we need to find thesolution at the receiver antennas, which is typically 5–15orders of magnitude smaller than the solution at thetransmitter antenna, and (e) we need to produce fastsimulations to enable inversion of the measurementsand/or a detailed numerical study of the simulationenvironment.

1.3 Fourier hp-Finite Element Method

To overcome the above challenges, we propose the useof a parallel Fourier hp-Finite Element Method (FEM),where h indicates the element size, and p the poly-nomial order of approximation, both varying locallythroughout the grid. Specifically, our method combinesa (1) 2D self-adaptive goal-oriented hp-FEM [7, 8,26, 27], which provides the geometrical flexibility of aFEM as well as high-accuracy simulations due to theexponential convergence in terms of the problem sizevs. the error in the quantity of interest (solution at thereceiver antennas), with (1) a 1D Fourier method forconsidering typical scenarios of marine CSEM prob-lems consisting of 2D subsurface resistivity models and3D finite-size EM sources. Thus, our method is suitablefor the simulation of objects with different scales (smallantennas and large layers), it performs simulations atnon-zero frequencies with possibly anisotropic forma-

tions (we solve full Maxwell’s equations), and it delivershigh-accuracy approximations of the solution at thereceiver antennas (quantity of interest). In addition,the entire implementation of our numerical methodis compatible with the use of sequential and parallelmachines in order to secure maximum performance.

The main contribution of this paper is the propercombination, extension, and application of a Fourierhp-Finite-Element Method [25, 30] to obtain highlyaccurate simulations of marine CSEM measurements.In addition, we describe a parallel implementation andcompare various solvers of linear equations, we validatethe method employing analytical solutions for simplemarine CSEM scenarios, and illustrate via numericalexperimentation the main physical principles behindmarine CSEM measurements and the main features ofour simulation method.

The method described in this paper provides highlyaccurate and reliable results within reasonable CPUtimes as it enables a significant reduction of the compu-tational complexity with respect to conventional sim-ulators without sacrifice of accuracy. In addition, themethod is suitable for simulation of resistivity loggingmeasurements, inverse problems, as well as for mul-tiphysics applications. Therefore, it aims to become awidely used simulation strategy for problems arising inthe oil industry.

The paper is organized as follows: Section 2 describesthe mathematical formulation of our Fourier FiniteElement Method, which we describe in detail alongwith the main implementation aspects on Section 3.Section 4 illustrates the performance of the method,including validation results and challenging simulationsof marine CSEM measurements. Section 5 is devotedtowards conclusions.

2 Formulation

In this section, we first introduce time-harmonicMaxwell’s equations, and the corresponding modelingof the sources and boundary conditions we utilize forsimulation of marine CSEM measurements. Then, wedescribe the corresponding 3D variational formulation.After taking a 1D Fourier transform, we derive a sim-ple variational formulation under the assumption ofconstant materials in the Fourier direction. We alsoindicate the steps needed to obtain a 3D variationalformulation under more general material distributions.Finally, we discuss the main differences between avariational formulation in terms of electric field E andmagnetic field H.

56 Comput Geosci (2011) 15:53–67

2.1 Time-harmonic Maxwell’s equations

Assuming a time-harmonic dependence of the forme jωt, with ω denoting angular frequency, Maxwell’sequations in linear media can be written as

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

∇ × H = (σ + jωε)E + Jimp Ampere’s Law,

∇ × E = − jωμ H − Mimp Faraday’s Law,

∇ · (εE) = ρe Gauss’ Law ofElectricity, and

∇ · (μH) = 0 Gauss’ Law ofMagnetism.

(1)

In the above equations, H and E denote the magneticand electric fields, respectively, real-valued tensorsε, μ, and σ stand for dielectric permittivity, magneticpermeability, and electrical conductivity of the media,respectively, ρe denotes the electric charge distribution,and Jimp, Mimp are representations for the prescribed,impressed electric and magnetic current sources,respectively.

For convenience, we introduce tensors◦μ := − jωμ

and◦σ := σ + jωε. We assume that det(

◦μ) �= 0, and

det(◦σ ) �= 0.

2.1.1 Closure of the computational domain

A variety of BCs can be imposed on the boundary ∂� ofa computational domain � such that the difference be-tween the solution of such a problem and the solution ofthe original problem defined over R

3 is small (see [11]).For example, it is possible to use an infinite elementtechnique [3], a Perfectly Matched Layer (PML) [24], aboundary element technique [10] or an absorbing BC.However, it is customary in geophysical applications toimpose a homogeneous Dirichlet BC on the boundaryof a sufficiently large computational domain, since theEM fields decay exponentially in the presence of lossymedia.

For simplicity, in the simulations presented below wewill follow the Dirichlet BC approach over a 2D cross-section of the domain, that is, we will impose n×E = 0on the boundary of the 2D cross-section � = ∂�.Specifically, we shall consider a domain of size 26 ×20 km. In the (third) spatial direction perpendicularto the plane of Fig. 1 (bottom), we will consider theoriginal infinite domain and we will apply a Fourier

transform method. We note that for the case of shallowwater, we need a very large computational domain inthe air layer or a more sophisticated BC (such as aPML).

2.1.2 Source and receiver antennas

In this work we model antennas by prescribing animpressed volume current Jimp or Mimp. Specifically,we inject a constant horizontal electric current over asmall box corresponding to the volume occupied by theelectrode. Receivers measure the average electric (ormagnetic) field over a small box corresponding to thevolume occupied by the receiver. Notice that we do notmodel sources/receivers as metallic objects, but ratheras small boxes with currents. We note that it is alsopossible to model antennas by using the equivalenceprinciple to replace the original impressed volume cur-rents with equivalent surface currents (see [13, 26] fordetails).

2.2 E-variational formulation

In this subsection, we describe our method in terms ofthe unknown electric field E. First, we define the L2-inner product of two (possibly complex- and vector-valued) functions f and g as:

⟨f, g

⟩

L2(�)=

∫

�

f∗gdV , (2)

where f∗ denotes the adjoint (conjugate transpose) offunction f.

By pre-multiplying both sides of Faraday’s law

by◦μ

−1, multiplying the resulting equation by ∇×F,

where F ∈ H�(curl; �) = {F ∈ H(curl; �) : (n×F)|� =0 } is an arbitrary test function, integrating over thedomain � ⊂ R

3 by parts, and applying Ampere’s law,we arrive at the following variational formulation afterincorporating the Dirichlet BC over � = ∂�:

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

Find E ∈ H�(curl; �) such that:⟨

∇×F ,◦μ

−1∇×E⟩

L2(�)

−⟨F ,

◦σE

⟩

L2(�)

= ⟨F, Jimp

⟩

L2(�)

+⟨

∇×F,◦μ

−1Mimp

⟩

L2(�)

∀F ∈ H�(curl;�).

(3)

Comput Geosci (2011) 15:53–67 57

2.2.1 Fourier transform

We define the unitary Fourier transform of a possiblycomplex- and vector-valued function f(x1) with respectto variable x1 as follows:

F r(f ) = F(f )(r) := 1√2π

∫

R

f(x1)e− jrx1 dx1 . (4)

The inverse Fourier transform is given by

f(x1) := 1√2π

∫

R

Fr(f )e jrx1 dr . (5)

Two important properties of the Fourier trans-form are its linearity and its compatibility withdifferentiation, namely if f(x1) is a differentiable func-tion, then Fr

(∂f∂x1

) = jrFr(f ). Defining operator ∇r :=(

jr, ∂∂x2

, ∂∂x3

), we have for f = ( f1, f2, f3):

∇r×(f1, f2, f3

) :=(

∂ f3

∂x2− ∂ f2

∂x3,

∂ f1

∂x3− jr f3, jr f2− ∂ f1

∂x2

)

.

(6)

Thus, we obtain the following identities:

Fr(∇×f

) = ∇r×(Fr(f )

), and

∇×(f2De jrx1

) = (∇r×f2D)

e jrx1 , (7)

where function f2D = f2D(x2, x3) is independent of x1.

2.2.2 2.5D Fourier f inite element formulation

In this subsection, we assume that material properties(namely,

◦σ and

◦μ) are constant with respect to x1, and

domain � can be expressed as � = �(x1, x2, x3) = R ×�2D(x2, x3). Similarly, we assume � = R × �1D(x2, x3).Using the definition of inverse Fourier transform withrespect to variable x1, we have:

E(x1, x2, x3

) = 1√2π

∫ ∞

−∞Fr(E)

(x2, x3

)e jrx1 dr . (8)

Employing a (mono-modal) test function of the formF(x1, x2, x3) := 1√

2πFrm(F)(x2, x3)e jrmx1 (without the in-

tegration), we obtain:⟨F,

◦σE

⟩

L2(�)

= 1√2π

⟨

Frm(F)e jrmx1,◦σ

1√2π

∫

R

Frn(E)e jrnx1 drn

⟩

L2(�)

.

(9)

Understanding the above integral in the distributionalsense, and using the fact that

1√2π

∫

R

e jrmx1 e− jrnx1 dx1 = √2πδrm,rn , (10)

where δrm,rn is the Kronecker’s delta function (equal to1 if rm = rn and 0 otherwise), we arrive at⟨F ,

◦σE

⟩

L2(�)=

⟨Frm(F) ,

◦σFrm(E)

⟩

L2(�2D). (11)

Similar expressions can be obtained for all terms of thevariational problem described by Eq. 3. Again, usingthe definition of inverse Fourier transform and Eqs. 6, 7and 10, we obtain:⟨

∇×F,◦μ

−1∇×E⟩

L2(�)

=⟨

∇rm×Frm(F),◦μ

−1∇rm×Frm(E)

⟩

L2(�2D)

,

⟨F, Jimp⟩

L2(�)= ⟨

Frm(F),Frm(Jimp)⟩

L2(�2D), and

⟨

∇×F,◦μ

−1Mimp

⟩

L2(�)

=⟨

∇rm×Frm(F),◦μ

−1Frm(Mimp)

⟩

L2(�2D)

. (12)

Thus, the variational problem (3) becomes

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

Find E= 1√2π

∫

R

Fr(E)e jrx1 dr, where for each r ∈ R:

Fr(E) ∈ H�1D

(curlr; �2D

), and

⟨

∇r×Fr(F) ,◦μ

−1∇r×Fr(E)

⟩

L2(�2D)

−⟨Fr(F) ,

◦σFr(E)

⟩

L2(�2D)

= ⟨Fr(F) , Fr(Jimp)

⟩

L2(�2D)

+⟨

∇r×Fr(F) ,◦μ

−1Fr(Mimp)

⟩

L2(�2D)

∀ Fr(F) ∈ H�1D(curlr; �2D) ,

(13)

where

H�1D

(curlr; �2D

)

= {Fr(F) ∈ H

(curlr; �2D

) : (n×Fr(F)) |�1D = 0}

,

H(curlr; �2D

)

= {Fr(F) ∈ L2(�2D) : ∇r×Fr(F ) ∈ L2(�2D)

}. (14)

Variational Formulation (13) is composed of a se-quence of independent 2D problems, a so-called 2.5Dproblem. The unknowns of each 2D problem are thethree components of the electric field. Each componentof the electric field depends upon two spatial vari-ables, namely, x2 and x3. To derive Formulation (13),we have assumed that material properties (

◦σ and

◦μ)

are constant with respect to variable x1. However, we

58 Comput Geosci (2011) 15:53–67

remark that material properties may incorporate anytype of anisotropy. This is a typical requirement for thesimulation of marine CSEM measurements.

2.2.3 3D Fourier f inite element formulation

For the case of general 3D materials, Eq. 11 andsubsequent formulas no longer hold, and it is neces-sary to perform a Fourier transform of the materialcoefficients. As a result, the final variational formula-tion is expressed as a sequence of coupled 2D problems.Specifically, each 2D problem couples with as many 2Dproblems as terms are needed to describe exactly thematerial coefficients in the Fourier transform.

2.3 H-variational formulation

To obtain a formulation in terms of magnetic field Hrather than in terms of electric field E, it is sufficient tofollow a similar approach to the one described in thispaper but by taking as the main equation the curl ofFaraday’s law rather than the curl of Ampere’s law.

In this work, we have implemented both formu-lations for comparison purposes. We note that BCsshould be consistent in order to compare results ob-tained with the E-formulation and the H-formulation.Specifically, the Dirichlet BCs used with the E-formulation in order to truncate the computationaldomain become Neumann BCs for the case of the H-formulation. Thus, the problem becomes singular (noDirichlet BC is present), and a Dirichlet node shouldbe introduced, as described in [7].

The simultaneous use of both E- and H-formulationscould be used for error analysis of the discretizationerror or the truncation error if we use non-consistentBC’s. A large discrepancy among both solutions indi-cates a large error, although unfortunately, the oppositeis not true in general.

3 Method: a Fourier hp-Finite Element Method

To solve variational problem (13), we first approximatethe inverse Fourier transform of electric field E by thefollowing Fourier series:

√2πE(x)=

∫

R

Fr(E)e jrxdr ≈M∑

n=−M

(rn+1−rn)Frn(E)e jrnx

≈ 2π

T

M∑

n=−M

F2πn/T(E)e jx2πn/T . (15)

In the above approximations, we have selected an in-tegration rule with all weights equal to 1, we havetruncated the infinite sum, and we have considereda uniform distance between integration points. Thisdistance is determined by parameter T. We note thatas M −→ ∞, the number of terms and computationalcost needed to properly approximate the above Fourierseries expansion increases. It is also well-known thatFourier series expansions are used to represent periodicfunctions. Thus, a finite value of T can be physicallyinterpreted as the presence of infinite sources at dis-tances T, 2T, 3T, etc. from the original source along theFourier direction. These additional undesired sourcesmay decrease the accuracy of the solution for small T.

To select a first approximation of T that can be lateradjusted based on the numerical results, we first assumethat the solution decays as (4πσd3)−1, where d is thedistance from the source to the receiver. This formulacorresponds to the exact solution of a point sourceoperating at zero frequency in a homogeneous medium.Under this assumption, we conclude that

Rel. Error (in %)

= 100 ·∣∣∣∣∣

n=∞∑

n=−∞

[1

4πσ((nT)2 + d2

)3/2

]

− 14πσd3

∣∣∣∣∣

/ 14πσd3

=∣∣∣∣∣∣

n=∞∑

n=−∞,n�=0

100((nT/d)2 + 1

)3/2

∣∣∣∣∣∣

=∣∣∣∣∣

n=∞∑

n=1

200(1 + (nα)2

)3/2

∣∣∣∣∣

, (16)

where α = T/d. Table 1 describes the behavior of theabove formula. We observe relative errors below 25%and 10% for values of α equal to 2 (T = 2d) and3 (T = 3d), respectively. An error level below 10%is acceptable and corresponds to what it is consid-ered an “accurate” solution in the context of marineCSEM measurements (note that a relative differencebelow 10% in Fig. 4 would be almost invisible to thehuman eye).

After selecting T, we truncate the infinite series (15).An approximation of the truncation error is numeri-cally evaluated by computing two (or four) additionalterms in the Fourier series expansion.

Each term Fr(E) of the infinite Fourier series ex-pansion (15) is computed using a 2D self-adaptive,goal-oriented, hp-FEM software that accurately sim-ulates problems involving coupled H(curl)- and H1-discretizations. For simplicity, we construct a uniquecommon hp-grid for all Fourier modes based on the

Comput Geosci (2011) 15:53–67 59

Table 1 Relative error (in percentage) according to Eq. 16 due to the use of a non-zero value of α = T/d

Values of α 1 2 3 4 6 10 100 1.e3 1.e4Rel. error (in %) 102 23 8 3.5 1.8 2.4e−1 2.4e−4 2.4e−7 2.4e−10

solution of the central Fourier mode. The goal-orientedhp-FEM delivers exponential convergence rates interms of the error in the quantity of interest versus thenumber of unknowns and CPU time. The outstandingperformance of the hp-FEM for simulating diverse re-sistivity logging measurements has been documentedin [25, 28, 29], and a similar performance is expectedto hold when simulating marine CSEM measurements.A detailed description of the hp-FE method and itsexponential convergence properties can be found in [7].We refer to [26] for technical details on the goal-oriented adaptive algorithm applied to simulation ofelectrodynamic problems.

3.1 Solver of linear equations

The choice of an adequate solver of linear equationsis critical for the efficient simulation of marine CSEMmeasurements. While iterative solvers are faster andutilize less memory than direct solvers (see Table 2),they also include a number of disadvantages. Namely,when solving a problem with M > 1 right-hand sides(as it occurs in inverse problems), the cost of itera-tive solvers rapidly increases, while the cost of directsolvers remains almost constant if M << N. In addi-tion, convergence properties of iterative solvers arehighly dependent upon the condition number of thematrix and the physics of the problem (e.g., specificiterative solvers are needed for EM problems in orderto avoid spurious modes). In particular, an iterativesolver requires the implementation of smoothers spe-cially designed to minimize the error of both the rota-tional and gradient parts of the solution, c.f. [2, 14]. Forsimplicity, and in order to avoid additional numericalerrors possibly introduced by the iterative solver, in thispaper we use a direct solver.

In the following, we compare the performance oftwo well known direct solvers of linear equations when

applied to a marine CSEM problem: 1) the parallelmulti-frontal solver MUMPS (version 4.8.3) [20]; and2) PARDISO [23]. Both solvers have been combinedwith the ordering of the unknowns provided by METIS(version 4.0) [19].

Figure 2 compares the performance—CPU time(left panel) and random access memory (RAM) (rightpanel)—of the sequential versions of PARDISO andMUMPS using both the in-core and out-of-core ver-sions of the solvers, when applied to the problemdescribed in Fig. 3. Results indicate that the out-of-core version of PARDISO is not competitive (utilizesroughly twice as more RAM and is twice slowest thanMUMPS out-of-core). Although the in-core versionof PARDISO is the fastest of all, in this work weutilize the out-of-core version of MUMPS due to itslow memory requirements and the possibility of usingit in distributed memory parallel machines (PARDISOonly supports shared memory parallel machines). Wenote that in Fig. 2 we could not solve the problemwith 3.2 million unknowns using PARDISO, becausethe solver stopped with an error indicating the lack ofavailable RAM.

Different performance of direct solvers can bedue to implementation aspects and/or better orderingof the unknowns. In the case of solvers PARDISOand MUMPS, both solvers use the ordering of un-knowns provided by METIS. Thus, it seems that in-corePARDISO is able to handle zeros in a more efficientway (reason why we observe less memory consump-tion), while the out-of-core implementation is clearlyless efficient than the one performed by MUMPS.Finally, we emphasize that due to the similar structureof the stiffness matrix for the marine CSEM problemsconsidered in this paper, we have observed almostidentical results for various CSEM problems in terms ofperformance of direct solvers, which we have omitted inthis paper to avoid repetition.

Table 2 Scalability of direct vs. iterative solvers

Scalability Direct solver (M right-hand sides) Iterative solver (M right-hand sides)CPU Time O(N2b) + O(NMb) LU fact. + Backward Subst. O(n2b) + O(MN b c) Preconditioner + Iter. SolutionMemory O(Nsb) O(Nb)

N is the problem size, b is the (average) bandwidth of the matrix, c is a constant that depends upon the condition number of the matrixand the choice of preconditioner, n is the size of the preconditioner, s is a constant that depends upon the structure of the matrix, andcan be as large as N, and M is the number of right-hand sides

60 Comput Geosci (2011) 15:53–67

15000 90000 540000 3200000

0.5

5

50

500

Number of Unknowns

CP

U T

ime

(s)

Comparison of Direct Solvers (Time)

MUMPS (In–core)MUMPS (Out–o–core)PARDISO (In–core)PARDISO (Out–of–core)

15000 90000 540000 320000010

100

1000

10000

Number of Unknowns

Mem

ory

(in M

B)

Comparison of Direct Solvers (Memory)

MUMPS (In–core)MUMPS (Out–of–core)PARDISO (In–core)PARDISO (Out–of–core)

Fig. 2 CPU time (left panel) and memory (right panel) usedby different direct solvers (analysis and LU factorization) vs.problem size when applied to a marine CSEM problem using

a Fourier hp-Finite Element Method. Tests performed on acomputer equipped with 32 GB of RAM and using only one coreof the available 2 GHz dual-core processor

3.2 Parallel implementation

Finally, we implement a parallel distributed-memoryversion of our hp-FEM to speedup computations andminimize the CPU time and memory used per proces-sor. Contrary to traditional parallelization strategiesbased on domain-decomposition schemes (em e.g.,[31]), in our implementation we store a copy of theentire computational grid in all processors. Decisionabout grid refinements is performed in parallel, whileactual refinements are executed in all processors si-multaneously, making unnecessary the so-called grid-reconciliation step intended to ensure the compatibilityof refinements occurring in different processors. Thus,the complexity of the parallel version of the software isdrastically simplified with respect to more traditionalparallelization strategies, while the overhead associ-ated with storing an entire copy of the 2D grid in all

processors remains at a level below 10% of the totalmemory and CPU time cost for a moderate number ofprocessors (below 200).

For the marine CSEM problems considered in thispaper, we observe an adequate performance up toa moderate number of processors (16–64). Then, aswe further increase the number of processors, a morerapid deterioration on the performance occurs due tothe overhead associated with our simple parallelizationstrategy.

4 Numerical results

In this section, we first verify our results by comparingthem against a semi-analytical software for a typical butsimple marine CSEM model. Then, we perform more

Fig. 3 Marine CSEMproblem composed of: a anair layer, b a 1,000-m thicklayer of sea-water withresistivity equal to 0.3 �·m,c a background material withresistivity equal to 1 �· m,d a 100-m thick (infinitelateral extend) oil-saturatedlayer with resistivity equal to100 �·m., e a transmitter(horizontal electric dipole)and f ten equally spacedreceivers. Distance betweentransmitter and receivers:from 1,000 to 10,000 m

Comput Geosci (2011) 15:53–67 61

challenging numerical simulations, which enables anunderstanding of the main physical principles govern-ing marine CSEM measurements. In all our models weassume that the relative magnetic permeability μ0 andrelative permittivity ε0 are equal to 1 over the entiredomain.

4.1 Verification

As illustrated in Fig. 3, we consider a marine CSEMproblem composed of a 100m-thick (infinite extend)oil-bearing layer in a background subsurface material,a 1,000-m thick layer of water, a transmitter located

0 2000 4000 6000 8000 1000010

10

10

10

10

10

10

10

Am

plitu

de o

f Ele

ctric

Fie

ld (

V/m

)

Horizontal Distance between TX and RX (m)

Without Oil, 0.25 Hz

1 MODE5 MODES9 MODESEXACT

0 2000 4000 6000 8000 1000010

10

10

10

10

10

Am

plitu

de o

f Ele

ctric

Fie

ld (

V/m

)

Horizontal Distance between TX and RX (m)

Without Oil, 1.25 Hz

1 MODE5 MODES9 MODESEXACT

0 2000 4000 6000 8000 1000010

10

10

10

10

10

Am

plitu

de o

f Ele

ctric

Fie

ld (

V/m

)

Horizontal Distance between TX and RX (m)

With Oil, 0.25 Hz

1 MODE5 MODES9 MODESEXACT

0 2000 4000 6000 8000 1000010

10

10

10

10

10

10

10

Am

plitu

de o

f Ele

ctric

Fie

ld (

V/m

)

Horizontal Distance between TX and RX (m)

With Oil, 1.25 Hz

1 MODE5 MODES9 MODESEXACT

–10 –10

–12

–14

–16

–18

–20

–10

–11

–12

–13

–14

–15

–16

–17

–11

–12

–13

–14

–15

–16

–17

–10

–11

–12

–13

–14

–15

Fig. 4 Amplitude of the electric field as a function of the horizon-tal distance between transmitter and receivers. Different curvesindicate different numbers of Fourier modes: a one mode (dotted

pink), b five modes (blue ’+’), c nine modes (black circles), andd exact solution (red solid line). Operating frequencies: 0.25 Hz(left panel) and 1.25 Hz (right panel)

62 Comput Geosci (2011) 15:53–67

Fig. 5 Marine CSEM problem composed of: a an air layer, b a1,000-m thick layer of sea-water with resistivity equal to 0.3 �·m,c a background material with anisotropic resistivity (1 �·m inthe horizontal direction and 3 �·m in the vertical direction), d

a 100-m thick (3,000-m lateral extend) oil-saturated layer withresistivity equal to 100 �·m., e a transmitter (horizontal electricdipole) and f ten equally spaced receivers. Distance betweentransmitter and receivers: from 1,000 to 10,000 m

50 m above the sea-floor, and ten equally spaced re-ceivers located on the sea-floor at horizontal distancesvarying from 1,000 to 10,000 m.

Figure 4 displays the amplitude of the electric fieldas a function of the horizontal distance between trans-mitter and receivers for the model problem of Fig. 3

Fig. 6 Final hp-grid for our model problem, assuming a lateralextent of the oil-saturated layer equal to 3,000 m, starting at a hor-izontal distance of 2,000 m away from the transmitter, and endingat 5,000 m away from the transmitter. Different panels corre-spond to different receiver locations, since we have constructedone optimal hp-grid for each receiver location. Specifically, we

display three grids corresponding to three receivers located athorizontal distances equal to 2,000 m (top-left), 5,000 m (top-right), and 8,000 m (bottom) away from the transmitter. Differentcolors indicate different polynomials orders of approximation,ranging from p = 1 to p = 8

Comput Geosci (2011) 15:53–67 63

(with and without the oil-saturated layer). The exactsolution for this problem has been computed usingthe analytical software EM1D (authored by ProfessorK.H. Lee, Lawrence Berkeley Labs, Personal Commu-nication, 2007) [32]. Numerical and analytical resultscoincide at both 0.25 and 1.25 Hz. Indeed, when usingonly five Fourier modes, the numerical solution andthe exact solution already coincide for all practicalpurposes.

From the physical point of view, we observe the ex-pected behavior that the decay of the solution becomesmore pronounced as frequency increases. We alsoobserve in Fig. 4 (top panel) the influence of the water-air interface in the response, which can be noticed atdistances above 7,000 and 5,000 m at 0.25 and 1.25 Hz,respectively. In addition, results of Fig. 4 (bottompanel) clearly indicate the presence of oil-bearing re-sistive layer, because the intensity of the electric field

at the receivers is larger than for the model without theoil-bearing layer (top panel).

4.2 Numerical simulations

Now, as described in Fig. 5, we consider a finiteoil-bearing layer embedded in a possibly anisotropicformation.

In order to illustrate the optimal 2D grids thatare automatically generated by the self-adaptive goal-oriented hp-FEM, we display in Fig. 6 the final hp-gridsdelivering an error in the quantity of interest below 1%for the model problem described in Fig. 5 assuming aformation with an isotropic resistivity equal to 1�·m.We emphasize that a 1% relative error in the quantityof interest provides a highly accurate solution for thecase of marine CSEM problems, since EM fields suffer

1000 8000 27000 64000 12500010

10

100

–1

–2

101

102

103

Number of Unknowns

Rel

ativ

e E

rror

(in

%)

hp–adaptivityh–adaptivityh–uniformp–uniform

Fig. 7 Top panel: Convergence history of h-uniform, p-uniform,goal-oriented h-adaptive, and goal-oriented hp-adaptive refine-ments for the central Fourier mode. Bottom panel: Final goal-oriented h- and hp-adaptive grids delivering 1% relative error in

the solution at the receiver, which is located 7,000 m away fromthe transmitter. Different colors indicate different polynomialsorders of approximation, ranging from p = 1 to p = 8

64 Comput Geosci (2011) 15:53–67

large attenuation as they travel from the transmitter tothe receivers.

Our goal-oriented grid-refinement strategy is basedon minimizing the error of the solution at the re-ceiver antennas by using the dual (adjoint) solution.We observe in Fig. 6 that most h-refinements occuraround both the transmitter and receiver electrodes,while most p-refinements take place in the subsurface.These refinements are consistent with the physics of theproblem, since the solution is known to be smooth onthe subsurface, with the most abrupt variations occur-ring around the transmitter and receivers electrodes.We remark that since we only consider structured initialhp-grids, there exist small elements used to reproducethe geometry of sources and receivers that still remainin the final hp-grid. However, we also observe newelements arising from local hp-refinements that areneeded in order to reduce the discretization error.

Figure 7 (top panel) describes the convergence his-tory of h-uniform, p-uniform, goal-oriented h-adaptive,and goal-oriented hp-adaptive refinements for the cen-tral Fourier mode when applied to the marine CSEMproblem described in Fig. 5 with the receiver located

7,000 m away from the transmitter and operating at0.75 Hz. We observe a dramatically faster convergencewhen using hp-adaptive finite element methods. Thecorresponding final goal-oriented h- and hp-adaptivegrids delivering 1% relative error in the solution at thereceivers are displayed in Fig. 7 (bottom panel).

Figure 8 (top panel) describes the vertical compo-nent of the magnetic field (Hz) for the direct (top-leftpanel) and dual (top-right) problems corresponding tothe marine CSEM problem described in Fig. 5 with thereceiver located 7,000 m away from the transmitter.Figure 8 (bottom panel) describes Hz of the directsolution multiplied by Hz of the dual solution, a quan-tity used to determine the Jacobian matrix needed forinversion.

Finally, Fig. 9 compares results for four differentmodels: (a) the model of Fig. 3 without the oil-bearinglayer, (b) the model of Fig. 3 (with an oil-bearing layer),(c) the model of Fig. 5 without anisotropy, and (a) themodel of Fig. 5 (with anisotropy).

From the results shown in Fig. 9 we draw the fol-lowing physical conclusions: First, it is essential to con-sider the effect of anisotropy in the formation because

Fig. 8 Top panel: Vertical component of Hz for the direct (top-left panel) and dual (top-right) problems corresponding to themarine CSEM problem described in Fig. 5 with the receiverlocated 7,000 m away from the transmitter. Bottom panel: Hz

for the direct problem multiplied by Hz for the dual problem,which is a quantity used to compute the Jacobian matrix neededfor gradient-based inversion methods. All the above quantitiesare displayed in logarithmic scale

Comput Geosci (2011) 15:53–67 65

0 2000 4000 6000 8000 1000010

–15

10–14

10–13

10–12

10–11

10–10

Am

plitu

de o

f Ele

ctric

Fie

ld (

V/m

)

Horizontal Distance between TX and RX (m)

0.25 Hz

NO OILINFINITE LAYER OILFINITE LAYER OILANISOTROPY

0 2000 4000 6000 8000 1000010

–18

10–17

10–16

10–15

10–14

10–13

10–12

10–11

10–10

Am

plitu

de o

f Ele

ctric

Fie

ld (

V/m

)

Horizontal Distance between TX and RX (m)

1.25 Hz

NO OILINFINITE LAYER OILFINITE LAYER OILANISOTROPY

0 2000 4000 6000 8000 10000–250

–200

–150

–100

–50

0

Pha

se o

f Ele

ctric

Fie

ld (

degr

ees)

Horizontal Distance between TX and RX (m)

0.25 Hz

NO OILINFINITE LAYER OILFINITE LAYER OILANISOTROPY

0 2000 4000 6000 8000 10000–600

–500

–400

–300

–200

–100

0P

hase

of E

lect

ric F

ield

(de

gree

s)

Horizontal Distance between TX and RX (m)

1.25 Hz

NO OILINFINITE LAYER OILFINITE LAYER OILANISOTROPY

Fig. 9 Amplitude (top panel) and phase (bottom panel) of theelectric field as a function of the horizontal distance betweentransmitter and receivers. Different curves indicate differentmodels: a without oil (dotted pink), b with an infinite oil-bearing

layer (blue ’+’), c with a oil-saturated finite layer (black circles),and d with a finite oil-bearing layer in an anisotropic formation.Operating frequencies: 0.25 Hz (left panel) and 1.25 Hz (rightpanel)

measurements are highly affected by it. Second, mea-surements obtained in the presence of an infinite oil-bearing layer are quite different from those obtained ina presence of a finite oil-bearing layer. Third, results aresensitive to the presence of the finite oil-bearing layer,and features such as its location and size can be derivedfrom the measurements.

The relative difference as we include the finite oil-bearing layer with respect to measurements without anoil-bearing layer are displayed in Fig. 10. These resultsillustrate the dependance of the measurements withrespect to the finite extent oil-bearing layer. Large rela-tive differences in the measurements are observed bothfor the amplitude and the phase. The largest differences

66 Comput Geosci (2011) 15:53–67

0 2000 4000 6000 8000 1000010

1

102

103

104

Horizontal Distance between TX and RX (m)

Rel

ativ

e D

iffer

ence

(in

%)

Relative Diff. of Amplitude

0.25 Hz0.75 Hz1.25 Hz

0 2000 4000 6000 8000 1000010

20

30

40

50

60

70

80

Horizontal Distance between TX and RX (m)

Rel

ativ

e D

iffer

ence

(in

%)

Relative Diff. of Phase

0.25 Hz0.75 Hz1.25 Hz

Fig. 10 Relative difference in (percentage) of the amplitude (left panel) and phase (right panel) of the results with a finite oil-bearinglayer with respect to the case without an oil-bearing layer. Different curves correspond to various frequencies, from 0.25 to 1.25 Hz

are localized around the location of the oil-bearinglayer.

All problems described in this Section have beensolved on a computer equipped with 32 GB of RAMand using only one core of the available 2 GHz dual-core processor. The amount of time needed for com-putations never exceeded half an hour per receiver (inmost cases only a few minutes was sufficient). The high-accuracy and limited computational resources used bythe method indicate its suitability for solving inversemarine CSEM problems (perhaps using the parallelversion for the case of inverse problems).

5 Conclusion

We introduced a Fourier hp-Finite Element formula-tion for the simulation of marine CSEM measurements.This formulation combines the use of higher-orderFEM with the use of small elements to efficientlyapproximate singularities. The formulation was imple-mented using efficient direct solvers of linear equa-tions, a self-adaptive goal-oriented refinement strategy,and a parallel implementation. This technology enableshighly accurate and efficient simulations of marineCSEM measurements in a very limited CPU time.

Simulation results were verified using simple modelsbased on a typical CSEM acquisition system in orderto illustrate the accuracy of the method. We showed

numerically that very limited number of Fourier modes(typically between five and nine) is enough to deliververy accurate simulations.

Additional simulations were performed to study andillustrate the main physical properties inherent to ma-rine CSEM measurements. In particular, we showedthe need for modeling electrically anisotropic forma-tions, and the ability of marine CSEM measurementsto detect finite oil-bearing layers.

Acknowledgements We are thankful to Dr. K.H. Lee, for pro-viding us with EM1D, a software used for computing the exactsolution for our model marine CSEM problem.

References

1. Alumbaugh, D.L., Newman, G.A.: Image appraisal for 2Dand 3D electromagnetic inversion. Geophysics 65, 1455–1467(2000)

2. Arnold, D.N., Falk, R.S., Winther, R.: Multigrid in H(div)and H(curl). Numer. Math. 85(2), 197–217 (2000)

3. Cecot, W., Rachowicz, W., Demkowicz, L.: An hp-adaptivefinite element method for electromagnetics. III: a three-dimensional infinite element for Maxwell’s equations. Int. J.Numer. Methods Eng. 57(7), 899–921 (2003)

4. Chave, A.D., Cox, C.S.: Controlled electromagnetic sourcesfor measuring electrical conductivity beneath the oceans. 1.Forward problem and model study. J. Geophys. Res. 87,5327–5338 (1982)

Comput Geosci (2011) 15:53–67 67

5. Christensen, N.B., Dodds, K.: 1D inversion and resolutionanalysis of marine CSEM data. Geophysics 72, WA27–WA38(2007)

6. Constable, S., Srnka, L.J.: An introduction to marinecontrolled-source electromagnetic methods for hydrocarbonexploration. Geophysics 72, WA3–WA12 (2007)

7. Demkowicz, L.: Computing with hp-adaptive finite elements.In: One and Two Dimensional Elliptic and Maxwell Prob-lems, vol. 1. Chapman and Hall (2006)

8. Demkowicz, L., Kurtz, J., Pardo, D., Paszynski, M.,Rachowicz, W., Zdunek, A.: Computing with hp-adaptivefinite elements. In: Frontiers: Three-Dimensional Elliptic andMaxwell Problems with Applications, vol. 2. Chapman andHall (2007)

9. Edwards, N.: Marine controlled source electromagnetics:principles, methodologies, future commercial applications.Surv. Geophys. 26, 675–700 (2005)

10. Gomez-Revuelto, I., Garcia-Castillo, L.E., Pardo, D.,Demkowicz, L.: A two-dimensional self-adaptive hp-finiteelement method for the analysis of open region problemsin electromagnetics. IEEE Trans. Magn. 43(4), 1337–1340(2007)

11. Gomez-Revuelto, I., Garcia-Castillo, L.E., Demkowicz, L.F.:A comparison between several mesh truncation methods forhp-adaptivity in electromagnetics. Invited paper to the spe-cial session “Numerical methods for solving maxwell equa-tions in the frequency domain”. Torino (Italia), September2007

12. Gribenko, A., Zhdanov, M.: Rigorous 3D inversion of marineCSEM data based on the integral equation method. Geo-physics 72, WA73–WA84 (2007)

13. Harrington, R.F.: Time-Harmonic Electromagnetic Fields.McGraw-Hill, New York (1961)

14. Hiptmair, R.: Multigrid method for Maxwell’s equations.SIAM J. Numer. Anal. 36(1), 204–225 (1998)

15. Hoversten, G.M., Chen, J., Gasperikova, E., Newman, G.A.:Integration of marine CSEM and seismic AVA data for reser-voir parameter estimation. SEG/Houston Annual Meeting,EM 3.4, 579–583 (2005)

16. Hoversten, G.M., Newman, G.A., Flanagan, G.: 3D modelingof a deepwater EM exploration survey. Geophysics 71, G239–G248 (2006)

17. Li, Y., Constable, S.: 2D marine controlled-source elec-tromagnetic modeling: part 2, the effect of bathymetry.Geophysics 72, WA63–WA71 (2007)

18. Li, Y., Key, K.: 2D marine controlled-source electromagneticmodeling: part 1, an adaptive finite-element algorithm. Geo-physics 72, WA51–WA62 (2007)

19. METIS: family of multilevel partitioning algorithms. http://glaros.dtc.umn.edu/gkhome/views/metis (2007). Accessed 30June 2010

20. MUMPS: A multifrontal massively parallel sparse directsolver (2010)

21. Newman, G.A., Commer, M., Carazzone, J.J.: ImagingCSEM data in the presence of electrical anisotropy. Geo-physics 75, F51–F61 (2010)

22. Orange, A., Key, K., Constable, S.: The feasibility of reservoirmonitoring using time-lapse marine CSEM. Geophysics 74,F21–F29 (2009)

23. PARDISO: Thread-safe solver of linear equations. http://www.pardiso-project.org/ (2008). Accessed 30 June 2010

24. Pardo, D., Demkowicz, L., Torres-Verdín, C., Michler, C.:PML enhanced with a self-adaptive goal-oriented hp finite-element method and applications to through-casing boreholeresistivity measurements. SIAM J. Sci. Comput. 30, 2948–2964 (2008)

25. Pardo, D., Demkowicz, L., Torres-Verdín, C., Paszynski,M.: Simulation of resistivity logging-while-drilling (LWD)measurements using a self-adaptive goal-oriented hp-finiteelement method. SIAM J. Appl. Math. 66, 2085–2106(2006)

26. Pardo, D., Demkowicz, L., Torres-Verdín, C., Paszynski,M.: A goal oriented hp-adaptive finite element strategywith electromagnetic applications. Part II: electrodynam-ics. Comput. Methods Appl. Mech. Eng. 196, 3585–3597(2007)

27. Pardo, D., Demkowicz, L., Torres-Verdín, C., Tabarovsky,L.: A goal-oriented hp-adaptive finite element method withelectromagnetic applications. Part I: electrostatics. Int. J.Numer. Methods Eng. 65, 1269–1309 (2006)

28. Pardo, D., Torres-Verdín, C., Demkowicz, L.: Simula-tion of multi-frequency borehole resistivity measurementsthrough metal casing using a goal-oriented hp-finite elementmethod. IEEE Trans. Geosci. Remote Sens. 44, 2125–2135(2006)

29. Pardo, D., Torres-Verdín, C., Demkowicz, L.: Feasibilitystudy for two-dimensional frequency dependent electromag-netic sensing through casing. Geophysics 72, F111–F118(2007)

30. Pardo, D., Torres-Verdín, C., Nam, M.J., Paszynski, M., Calo,V.M.: Fourier series expansion in a non-orthogonal system ofcoordinates for simulation of 3D alternating current boreholeresistivity measurements. Comput. Methods Appl. Mech.Eng. 197, 3836–3849 (2008)

31. Paszynski, M., Kurtz, J., Demkowicz, L.: Parallel fully auto-matic hp-adaptive 2D finite element package. Comput. Meth-ods Appl. Mech. Eng. 195(7-8), 711–745 (2006)

32. Song, Y., Kim, H.J., Lee, K.H.: High frequency electro-magnetic impedance for subsurface imaging. Geophysics 67,501–510 (2002)

33. Tompkins, M.J., Srnka, L.J.: Marine controlled-source elec-tromagnetic methods: introduction. Geophysics 72, WA1–WA2 (2007)

34. Um, E.S., Alumbaugh, D.L.: On the physics of the marinecontrolled-source electromagnetic method. Geophysics 72,WA13–WA26 (2007)