the square kilometre array - engineers australia...csp is lead by nrc canada with mda managing...

The Square Kilometre Array

CSIRO ASTRONOMY AND SPACE SCIENCE

John Bunton | CSP System Engineer

6-8 March 2013

SKA1 Low antenna station (Australia)

Square Kilometre Array

SKA1 MID antennas (South Africa)

SKA1 Overview

SKA1-low stations include Station Beamformer

CSP includes Correlator, Tied Array beamformers, Pulsar Search Engine and Pulsar Timing Engine

I work here

My work

The Low Correlator and Beamformer (CSP_Low.CBF) consortium are part of the larger Central Signal Processing (CSP) Consortium

CSP is lead by NRC Canada with MDA managing

CSP_Low.CBF is lead by CSIRO (Australia)

With ASTRON (Netherlands) and

AUT (New Zealand) as collaborator

SKA Science



Rainer Beck

Radio Telescope Sensitivity


Ideal Antenna and Receiver

Output power proportional to Temperature

27C =300K

-270C =3K

Power=1 Power=100


A real system

A real receiver has internal noise

Model as a resistor with an equivalent temperature Tr

Total power Tsky + Tr

Want to minimise Tr

A good system Tr=20K

Tr

Tsky

Signal = Tsky + Tr


Power in a resistor

Power = kTB k Boltzmann’s Constant = 1.38 x10-23 Joules/Kelvin

T Temperature of the resistor (20 Kelvin)

B Bandwidth

For 1Hz bandwidth, 20K system temperature Power at receiver input approx. 2.8 x 10-22 W

For a 1 MHz bandwidth its 2.8 x 10-16 W

A 1W transmitter at 100 km is ~10-11 W/m2


The Jansky

15 m antenna, area 177 m2, and after normalising for bandwidth

Equivalent incident power = 1.6 x 10-24 W/m2Hz

This is an inconveniently small quantity

So Astronomers invented the Jansky

1 Jansky = 10-26 W/m2Hz

Our antenna needs a 160 Jansky source for the power to double at the output.

Or a 1 nanoW transmitter at 100 km with at a 1MHz bandwidth


Antenna Signal to Noise

A strong source (radio star) is 10 Jansky Signal to Noise Ratio 0.06, -12dB (note our signal is less than unwanted noise)

In the 80s I worked for the Fleurs Radio Telescope just outside Sydney It could detect sources of ~1 mJansky

SNR 0.000 006 = -52dB

ASKAP can reach ~10 μJansky, SNR = -72dB (10-7.2)

SKA less than 1 μJansky, SNR less than -82dB (~1:1billion)


Interferometry

How the SKA work


The Interferometer

The SKA uses interferometers

Signals from pairs of antennas are Correlated

A correlations is a multiply and accumulate. We average to improve signal to

noise ratio

The conjugate * cancels phase offset

Signals from star are in-phase

u (wavelengths)

q

X

++

*


Drunken Walk

Consider a drunk who is trying to get home.

He cab stagger from one lamp post to the next.

But when he gets there he can’t remember which way to go. So each time he sets of in a random direction

After N lamp posts how far, on average, has he gone Answer √N

After 100 lamp post he has gone about 10


The Correlator

The signal from each antenna has two component The sky signal (wanted)

Antenna (System) noise (unwanted)

The system noise is randomly in-phase and out-of-phase. The signal adds in the same manner as a drunken walk - grows as √N

The sky signal is maintained in phase – grows proportional to N

Improvement ~√N

X

++

*


How good does it get

For the SKA bandwidths as much as 5 GHz 5 x 109 samples per second

Observations can be more than 24 hours 105 seconds

Number of sample = Bτ B Bandwidth, τ Time = 5 x 1014

SNR improvement √Bτ ~ 2 x 107 (73dB) For a single pair of antennas.

SKA has 20,000 (Mid) and 256,000 (Low) pairs of antennas


Warning: Complex Arithmetic coming up

Phase and phase difference are important

Product of two real sinusoids

cos(ωt) cos(θ ) = (cos(ωt + θ) + cos(ωt - θ))/2 Yuk!

Complex sinusoids

ejωt = cos(ωt) + j sin(ωt) j= √-1

think real in-phase, imaginary (j term) quadrature phase

Now product of two complex sinusoids

ejωt ejθ = ej (ωt + θ) Easy


Star brightness B at position l = cos(q)

(direction cosine)Distance between two antennas u

in wavelengts

Extra path length to one antenna of

D= u cos(q) = ul

Adds a phase term of ej2πul

Interferometry 101


uljBeoutput 2

u co

s(q)

u (wavelengths)

q

Complex

Correlator

Consider real part of output

= Bcos(2πul)

Cosine variation as θ varies l = cos(q) and change in θ small

Interferometer is sensitive to a “spatial” frequency A component of the sky image with a

sinusoidal variation across the sky

Interferometry 101


u (wavelengths)

q

X

++ Dump Sum once a second

q Period proportional to1/u

*

uljBeoutput 2

Summing over all l we find

If we measure at all separation we measure all “spatial” frequencies

We have measured the spectrum of the image

Measure different “frequencies” by having antennas at different spacings

Fourier transform “spectrum” to get image

Interferometry 101


uljBeoutput 2

dlelBuoutput ulj

2)()(

u (wavelengths)

q

X

++ Dump Sum once a second

q Period proportional to1/u

*

Correlation between pairs of antennas generate spatial frequency (Fourier) components of the image

N antennas gives N(N-1)/2 spacings (usually incomplete)

Transform to give (dirty) image (further processing needed)

Synthesis Mapping


0

20

40

0

20

400

0.5

1

300 400 500 600 700

200

300

400

500

Antenna location Fourier Components Image of Point Source

-200 0 200 400

-200

0

200

400

Musical Analogy


Fourier Components

-200 0 200 400

-200

0

200

400

Music score defines the notes (frequencies) and their position (time)

Fourier components defines spatial frequency and also a position (in this case and angle)

Need and orchestra to process the score into music

Need a complex computer program to convert Fourier component to an image

Spectral Response

Many radio astronomy signal are due to atomic transition – Neutral Hydrogen at 1.42 GHz, OH masers at 1.665 GHz…

Want spectrum of signal – Colour image, not just black and white

EITHER Measure correlation with different delays– Cross Correlation function and Fourier Transform – XF correlator

OR Split signal into subbands and measure correlation in each– Cross Power Spectrum - FX correlator

X

A

D

X

A

D

X

A

D

X

A

D

X

A

D

X

A

D

X

A

nD

Antenna 1 Antenna 2 Antenna 1

Frequency

Transform

Shift R

egis

ter

Shift R

egis

ter

Accumulator

Shift Register +x

Note: no change in

sample rate

Antenna 2

Frequency

Transform

Shift R

egis

ter

Shift R

egis

ter


Which to Use for the SKA

XF – complex multiplies per input sample = No. Channels ~104

FX – complex multiplies per input sample = 1 Plus overhead for frequency transform ~12 per complex antenna per sample

But number of correlations is N(N-1)/2 where N is number of antennas

FX computation more efficient when No. Channels > 1 + 24/N

X

A

D

X

A

D

X

A

D

X

A

D

X

A

D

X

A

D

X

A

nD

Antenna 1 Antenna 2 Antenna 1

Frequency

Transform

Shift R

egis

ter

Shift R

egis

ter

Accumulator

Shift Register +x

Note: no change in

sample rate

Antenna 2

Frequency

Transform

Shift R

egis

ter

Shift R

egis

ter

XF FX


Each antenna generates signal for two polarisations– X,Y linear or left and right circular

Must form all combination X1 X2*, Y1 Y2

*, X1 Y2*, Y1 X2

*,– Stokes parameters

Compute load for a single pair of antennas 4 multiplies and 4 adds per complex multiply

4 complex multiplies per data sample

1 data sample per Hz (complex data)

32 Gflops per GHz of bandwidth

For correlation (excluding image processing, beamforming etc)

• SKA Low 1.26 Pflops

• SKA Mid 2.95 Pflops

Polarisation and compute load


Filterbanks

From 1GHz to 1kHz


Generalised Filterbank

I/Q mixer on each channel, selects band centre

Filter to required bandwidth (impulse response ho(n), ho(t) analogue)

Down sample to required sample rate


Down convert

or ADC if

mixer and

filter

analogue

IQ Mixer

Low Pass filter

Cos(ωt)

Sin(ωt)

Single output from filterbank

Multiplication by complex sinusoid and impulse response can be interchanged.

Let w=e-jω


w0.n

Sum

w1.n

wm.n

x(n)

Xo

X1

XN-1

ho(n)

ho(n)

Sum

Sum

Single output, simplified

Need to do multiplication of input by ho(n) once

Take block of input data and value by value multiply x(n).ho(n)

On the right is a Fourier Transform

Choose w and m appropriately Have a Fast Fourier Transform


Fourier

Transform

Pre Multiply

Decimating the FFT

On 2GS/s data a 1MHz bandwidth filter and ho(n) ~20,000 samples long.

Therefore the channel spacing is 0.1MHz

We only want to keep every tenth output from the FFT. The rest can be discarded.

So why calculate them at all? Have

Want ever rth output:


1

0

)/2().().()(

N

n

nkNjenxnhkX

1

0

')/2().().()'(

N

n

nkNrjenxnhkX

Apologies for the Maths

If the length of the original impulse response and FFT is rM then

Interchanging the order of summation gives (ALMA memo 447)


1

0

')/2(1

0

1

0

'..2')/2(1

0

1

0

'))(/2(1

0

).().(

).().(

).().()'(

M

n

nkMrjr

m

M

n

kmjnkNrjr

m

M

n

kmMnNrjr

m

emMnxmMnh

eemMnxmMnh

emMnxmMnhkX

')/2(1

0

1

0

1

0

')/2(1

0

.)().(

).().()'(

nkMjM

n

r

m

M

n

nkMjr

m

emMnxmMnh

emMnxmMnhkX

Reduced size (one rth)

Fourier transform

Example r = 2

Consider an 8 point Fourier Transform and only every second output wanted

Delete every second row.

What is left is two 4-point Fourier Transforms

Sum the data before the transform and do one Transform


6

4

2

0

7

6

5

4

3

2

1

0

49423528

2460

35302520

4040

21147

246

15105

404

21181512

6420

7654

963

642

321

1

1

1

1

1111

1

1

1

1111

f

f

f

f

x

x

x

x

x

x

x

x

wwww

wwww

wwww

wwww

www

www

www

www

wwww

wwww

wwww

www

www

www Note shaded

blocks are the

same after

deletions


Graphical representation

Multiply input by prototype filter

Segment into r section of length M

Sum the r section to give output of length M

Fourier Transform

Generates a single output for each frequency channel

M-Point

FFT

M samples

Segment and sum

x

Filter h0(n)

Data

Overlapped

filter

segmentsFilter bank output


Critically sampled filterbank

Want to continuously process the data

Next output obtained by shifting the data) and applying the same process.

If the shift is by M samples then each input to the FFT is processing largely the same data. Gives a simple architecture

M-Point

FFT

M samples

Segment and sum

x

Filter h0(n)

Data

Overlapped

filter

segmentsFilter bank output

M-Point

FFT

po(m)

p1(m)

pM-1(m)

Pre-summation

Implementation of Critically Sampled Filterbank

SKA1 Low

to be built in Australia


SKA1 Low Location

Geraldton

Boolardy

Regulation of RFI

Geraldton

Low Population Density

gazetted towns: 0

population: “up to 160”

Geraldton

Australian Strengths

gazetted towns: 0

population: “up to 160”

Geraldton

SKA1 Low Station processing (Europe)

Bandwidth 300 MHz (Sky frequency 50-350 MHz)

512 antenna stations

Each station 256 dual polarisation antenna (log periodic)

Digitise 262k signal at 0.8 GS/s – total of 105 THz of bandwidth

Beamform to produce a 300MHz beam– Frequency domain processing.

– Apply filterbank 1024 channels (800Mhz to 781kHz)

– Beamforming in a single channels is just

– Multiply each input by a ejθ, adjust phase, and add

6 Petaflops

Total power ~1MW

SKA1 Low Correlator Base Requirements(CSIRO, ASTRON, AUT) Bandwidth 300 MHz (Sky frequency 50-350 MHz) From low station as 384 channels of 0.781kHz each

Full Stokes

512 antenna stations

Correlator compute load 1.25 Pflops equivalent About order of magnitude more than JVLA correlator (US biggest current)

About half of the proposed SKA1 Mid correlator

Basic operating mode is a “zoom mode” with at least 64k frequency channels –across 300MHz, ~172 per 781 kHz beamformer channel Resolution 4.07 KHz

Plus Zoom Modes

Four Independent Zoom Bands nominally either 4 MHz

8 MHz

16 MHz or

32 MHz each

– 256 possible combination

At least 16k channels in each zoom band

In addition “continuum” required at frequency within 300 MHz observing band, but not in a zoom band

Plus Subarraying

The telescope can operate as 1 to 16 INDEPENDENT subarrays

No antenna station can belong to the same subarray

Plus provision is made for a maintenance (17th) subarray

Up to 512 antennas when there is a single subarray

Each subarray operates independently

Last implies we cannot change firmware to change modes. FPGA based correlator

Additional outputs from SKA1_Low.CBF

16 voltage beams full bandwidth for pulsar timing

500 pulsar search beams at 128MHz bandwidth

Pulsar search can trade beams for bandwidth– 500 beams 128 MHz,

– 250 beams 2x128 MHz, or

– 133 beams 3x128 MHz

Again frequency domain beamforming.

Plus Multibeaming

Each subarray can apportion its 300MHz to 8 beams

Each beam is a contiguous section of bandwidth

Each beam pointing is independent of the other beams

Spectrum of beams can overlap Example 8 beams all covering 200-237.5 MHz on the sky

Large number of mode

“Normal” observing with zoom modes, subarraying, multibeaming, and independent scheduling blocks.

In the following will show how these are implemented in SKA1_Low Correlator and beamformer

Implementing Delay Correction

Must bring all signal to same wavefront before correlation in effect add delay of u.cos(θ) to second antenna

u co

s(q)

u (wavelengths)

q

Complex

Correlator

Step 1 Remove bulk delay by delay sampled data (~1us accuracy for LOW). Left with fractional delay error of up to 0.5us across ~1MHz

delay = -dθ/dω (dω.delay = 180 degrees)Phase changes by up to 180 deg across ~1MHz

Fine Delay Correction with Fractional Time Delay filters

FIR filter with values sampled from a continuous time filter.

Change the initial sampling point of continuous time filter (red + and blue dot are a half sample different in delay

Problems Residual amplitude and phase errors, possibly

delay errors

– Changes with delay value

– Improve by making filter longer

In practice filter length large – more compute intensive than a filterbank

Filter Impulse Response

Fine Delay Correction with Phase Slope

In frequency domain delay is equivalent of a phase slope

“Normal” frequency resolution of correlator requires ~256 channel filterbank. Phase slope across a fine channel (4 kHz) is less than 1 degree (±0.5deg), average error is zero

Apply correct phase, as a function of time, to each fine channel

Problem: Antennas with different velocities = different rate of change of delay = Doppler Shift, Different Doppler shifts on different antennas (less than 10Hz)

Fine channels select different part of the sky spectrum – correlation loss

Solution apply Doppler correction before fine filterbank

Implementing Spectral Resolution

Six spectral resolution required:– Continuum, “normal” (4kHz) and four zoom modes

Solution 1 Five Separate filterbanks– Implement a separate filterbank for each resolution – No filterbank for Continuum so fine delay must be fractional delay filter

Solution 2 Combine zoom filterbanks– Zoom filterbanks are power of 2. Build variable length filterbank – For data from stations Normal mode is resolution not a power of 2, separate

filterbank– Have two filterbanks and fractional delay filter

Solution 3 Frequency averaging– Implement filterbank at finest frequency resolution and average in frequency to

achieve other resolutions– 4096 channel filterbank -226Hz resolution,– Average 1, 2, 4, 8 channels gives 226, 452, 904Hz, 1.8kHz - all zoom modes– Average 18 channels give “normal” observing of 4kHz, average 3456 channels gives

continuum (781kHz)

SKA_Low solution

Use frequency averaging to implement all resolutions A single filterbank,

Uniform data flow into correlator (same for all resolutions)

Use phase slope method for fractional delay 4096 channel filterbank = 0.02degree maximum phase error across channel

Very small phase error

No added amplitude error

Small compute load

Correlation – six frequency resolutions

Correlator 1.25PFlops require 192 FPGAs

Data output from filterbank FPGAs on 8 x 25Gbps links

One link to group of 24 – cross connect with group of 24

Each FPGA process 1.56 MHz

Frequency averaging

for different frequency res.

24-way cross

connect

Correlator Frequency

Accumulation

Correlations

to SDPGearbox

Visibility Sharing &

Packetisation

Passive

Optical

Backplane

Passive

Optical

Backplane

From

Station

FPGAs

HMC

Memory

HMC

Memory

Multibeaming

For correlator difference in beam is simply a different delay polynomial (description of delay as a function of time)

Each antenna station is in a single subarray.

Eight beam per subarray require 8 delay polynomials for each antenna.

For each coarse channel apply the appropriate delay polynomial.

All extra complexity is with the filterbanks

Correlator independent

Subarraying

If there is NO SUBARRAYING correlation between all pairs of antennas is performed.

In this design all correlation for single frequency channel occurs in a single FPGA and are stored to DRAM

A subarray selects a subset of all possible correlation

Arrange data in DRAM so that a block read are for correlation where one the of antenna station inputs is common

SUBARRAYING becomes For each antenna station in a subarray read blocks for that antenna station

Select the subset of correlation for the subarray and transmit for data processing

Hardware Proposal

CSIRO-ASTRON Colaboration


Hardware implementationGemini Processing board

CSIRO ASTRON(Netherlands) collaboration

Optical Transceiver and FPGA SERDES number now allow

FULL OPTICAL DATA CONNECTIONS

Proposed processing board is a simple single board unit with

• A single FPGA

• Parallel Optical transceivers for I/O

(48 fibres total)

• Four HMC for data storage

• SFP+ 10GE for Monitor&Control


Perentie Rack Unit

Packaging

• Four Gemini processing boards

• Liquid cooling

• COTS Power

• Optical to Front Panel

• In a standard 1U rack unit

• 32Tflop equivalent

• 48 Rack units for correlation


Interconnecting Perentie Rack Units

If Front Panel had single fibres simply route each fibre to destination

But to keep density down have multi-fibre ribbons (up to 24/ri)

COTS Passive Optical Backplanes connect a single fibre from one ribbon to another. User writes specifications and has it built. Similar to acquiring circuit boards.


Putting it all together


Thank youCASSJohn BuntonSKA1 CSP System Engineer

t +61 2 9372 4420e [email protected] www.atnf.csiro.au/projects/askap

PO BOX 76 EPPING, 1710, AUSTRALIA

Example ASKAP

Major Cross connects for SKA1_Survey Filterbank to Beamformer ~2Tb/s per antenna

Beamformer to Correlator ~100Tb/s


the square kilometre array - engineers australia...csp is lead by nrc canada with mda managing...

Documents