the expanded very large array...5.13118 4.02882 cage1 2.125 cage2 2.775 3 4 2.450 cage5 sw1 1 2 3...

National Research CouncilCanada

Conseil national de recherchesCanada

The EVLA CorrelatorP. Dewdney

Herzberg Institute of AstrophysicsNational Research Council Canada

P. Dewdney EVLA Review - May 2006 2National Research CouncilCanada


Outline

1. Correlator Performance Goals2. Hardware Progress3. Software Progress4. System Progress5. Prototype Testing6. Installation Estimates7. Funding, Budget & Schedule8. Risk Issues



Key EVLA Processing Capabilities8 GHz Bandwidth (dual polarization).Full polarization processing.Wide-field imaging.

Deep ImagingPolarization

16,000 channels at max. bandwidth (BW).>106 channels at narrow BWs.Spectral resolution to match any linewidth.Spectral polarization (Zeeman Splitting).

Narrow spectral linesWideband searches

8 tunable 2 GHz wide bands.Each band - 16 tunable sub-bands.Sub-band – independent spectral resolution.Simultaneous line and continuum.

Flexibility Many resources

1000 pulsar “phase bins”.“Single-dish” data output to user instruments.Very fast time sampling (<20 µs).

High time resolution

EVLA Correlator System Diagram

Significant Events Since Dec. 2004

• Correlator Chip CDR – Jan/05• Software Review – Jan/05.• Correlator Preliminary Design Review – July/05.• PCB fabricator contract signed – Jan/06.• Prototype Correlator chip wafers fab’d – Feb/06.• Software Review – Mar/06.• Prototype chip delivery – early Jun/06.• Baseline Board prototype delivery – Jun/06.• Station Board prototype delivery – est. Aug/06.



Hardware Progress• FPGA’s

– All FPGA’s are designed, including the Filter Chip.• Station board

– Layout is close to completion, including Design-for-Manufacture approval.– Presently undergoing signal integrity simulation.– Prototype fabrication order expected in June.

• Baseline Board– Prototype fabrication under way.

• Phasing board– Draft specification (RFS) released.– Deferred in time but not reduced in priority.

• Other boards– Prototypes for all other boards already fabricated.– Fan-out board needs redesign for better signal margins.

• System– Racks designed and prototyped; thermal analysis done.– Power system RFP draft written.– Reliability analysis system in place – first draft of analysis complete.

Station Board Layout

• “Daughter” Board - brown.

• Power Supplies – pink.

• FPGA’s – Green

• Connectors – light brown and white.

• ~5000 parts.

• 140 required for EVLA.

Size: ~510 x ~410 mm

Baseline Board Layout

• Green chips – front side.

• Blue chips – back side.

• 8 x 8 array of correlator chips.

• LTA chips on the back side.

• Recirculation Controller.

• ~12000 parts

• ~177 required for EVLA.

0.0000.118

19.68719.569

9.8435

0.413

0.000

.2185

19.4685

19.687

3.5205

1.27914

3.3265

10.913

11.77914

13.8265

6 x .106" DIA

J3_1

J3

17.418

15.839

14.260

12.681

11.102

9.523

7.944

6.365

5.61295

4.580

8.3377

7.5503

5.9713

6.7587

9.1293

9.9167

10.7083

11.4957

12.2873

13.0747

13.8663

14.6537

15.4453

16.2327

17.0243

17.8117

5.13118

4.02882

CAGE1

CAGE22.125

2.775 CAGE3

CAGE4

2.450 CAGE5

SW1123

SW2

3

2

1

U100

U425 U109 U113 U427 U110 U428

U1_7

U2_7

U1_6

U2_6

U1_5

U2_5

U1_4

U2_4

U1_3

U2_3

U1_2

U2_2

U1_1

U2_1

U1

U2

U1_63

U2_63U3_63

U1_62

U2_62

U1_61

U2_61

U1_60

U2_60

U1_59

U2_59

U1_58

U2_58

U1_57

U2_57

U1_56

U2_56

U1_55

U2_55U3_55

U1_54

U2_54

U1_53

U2_53

U1_52

U2_52

U1_51

U2_51

U1_50

U2_50

U1_49

U2_49

U1_48

U2_48

U1_47

U2_47U3_47

U1_46

U2_46

U1_45

U2_45

U1_44

U2_44

U1_43

U2_43

U1_42

U2_42

U1_41

U2_41

U1_40

U2_40

U1_39

U2_39U3_39

U1_38

U2_38

U1_37

U2_37

U1_36

U2_36

U1_35

U2_35

U1_34

U2_34

U1_33

U2_33

U1_32

U2_32

U1_31

U2_31U3_31

U1_30

U2_30

U1_29

U2_29

U1_28

U2_28

U1_27

U2_27

U1_26

U2_26

U1_25

U2_25

U1_24

U2_24

U1_23

U2_23U3_23

U1_22

U2_22

U1_21

U2_21

U1_20

U2_20

U1_19

U2_19

U1_18

U2_18

U1_17

U2_17

U1_16

U2_16

U1_15

U2_15U3_15

U1_14

U2_14

U1_13

U2_13

U1_12

U2_12

U1_11

U2_11

U1_10

U2_10

U1_9

U2_9

U1_8

U2_8

J15

J12

J11

1.216

5.006

18.520

19.035

16.6285

15.0495

17.9298

16.3508

17.3647

16.8597

15.2807

15.7857

D3

D2

D1

D17

D4

D35

D36

D37

D5

D6

D10

D12

D13

D14

D15

D16

13.7017

12.6277

12.1227

U105

U107

U429

U111

U108

U115

3.150

14.2067

10.5437

9.4697

8.9647

11.0487

7.3857

6.3117

5.8067

7.8907

.320

U4_7

U5_29

U5_30

U5_31

U5_28

U4_6

U5_25

U5_26

U5_27

U5_24

U4_5

U5_21

U5_22

U5_23

U5_20

U4_4

U5_17

U5_18

U5_19

U5_16

U4_3

U5_13

U5_14

U5_15

U5_12

U4_2

U5_9

U5_10

U5_11

U5_8

U4_1

U5_5

U5_6

U5_7

U5_4

U4

U5_1

U5_2

U5_3

U5

6.993196.87681

6.274

6.027

18.04619

U3_7 U3_6

U3_62

U3_54

U3_46

U3_38

U3_30

U3_22

U3_14

U3_5

U3_61

U3_53

U3_45

U3_37

U3_29

U3_21

U3_13

U3_4

U3_60

U3_52

U3_44

U3_36

U3_28

U3_20

U3_12

U3_3

U3_59

U3_51

U3_43

U3_35

U3_27

U3_19

U3_11

U3_2

U3_58

U3_50

U3_42

U3_34

U3_26

U3_18

U3_10

U3_1

U3_57

U3_49

U3_41

U3_33

U3_25

U3_17

U3_9

U3_56

U3_48

U3_40

U3_32

U3_24

U3_16

U3_8



Software Progress• GUI-based prototype testing software

– Complete enough to be useful already.– All FPGA’s covered except Phasing Board.– Provides “engineer’s view of correlator system.

• Permanent maintenance value.– Top-level GUI’s now under development.

• Virtual Correlator Interface– Well defined except for correlator output area.– Communications protocol well defined except for transport layer.

• Master Correlator Control Computer (MCCC)– Work deferred for GUI development.– Architecture/scope well defined.

• Real-time control software– “CMIB” processor on each board.– Operating system has been working for a long time.– XML-based communication with “outside world”.– Drivers for FPGA’s are well under way.

Correlator Software System Context

Archive

Model Server

Backend

Observation Layer

Log Server

Station Board Baseline Board

MCCC(1+1)

CPCC(1+1)

Monitor & Control Data

Astronomical (observed) Data

EVLA Monitor & Control

Correlator System GUI WIDAR Correlator - Top level system view (40 stations configuration)

Main

Rack 006 Rack 010

Rack 001 Rack 004Rack 003Rack 002 Rack 005

Rack 008 Rack 009

Rack 105 Rack 109

Rack 100 Rack 103Rack 102Rack 101

Rack 106

Rack 104

Rack 107 Rack 008

Rack 115

Rack 110 Rack 113Rack 112Rack 111 Rack 114

Rack 007

MCCC 1Active

MCCC 2Standby

CPCC 2Active

CPCC 1Standby

BootServerLog Server

2007-123-09:30:00.000 In progress2007-123:10:35:30.0002007-123-09:45:34.0002007-124:11:05:00.0002007-122:23:30:00.0002007-123:10:15:00.0002007-123:09:30:00.000

In progressRejectedIn progressAcceptedIn progress

Status

Test21V01V23V01V22V014Q01V021A90V013Q23V013Q22

Observation Start Time

Accepted

PowerSystems

Heatingand

Cooling

Details

01 4020304050506

44222

001-0-0001-0-4001-1-0001-1-4002-0-0002-0-2002-0-4

001-0-1001-0-5001-1-1001-1-5002-0-1002-0-3002-0-5

Board2Board1#Boards

EVLA07EVLA06EVLA05EVLA04EVLA03EVLA02EVLA01

Antenna Quad

4

Backend101-1-7 110-1-7Running

2007-123-09:45:34.010 2007-123-09:45:34.010

TGM 1

Time 2Time 1Time 0StatusBoard

TGM 0

Running

Details

Individual RackWIDAR Correlator - Rack

Main

31 2

Crate 0

0 4 5 76

31 2

Crate 1

0 4 5 76

Rack 001

---1514 ---

OK 123.23.1.60------------------

TGMTGM123.23.1.60

131211109877 OK

OK 123.23.1.1OKOK

InitializingOK------

123.23.1.17123.23.1.25123.23.1.34

STBSTBSTBSTBBLB

TypeIP

6543210

Board Status

123.23.1.9

2007-90-11:15 Replaced STB12007-89-13:45 Initial installation

Operator Log

Station Board Top Level

Station Board Rack: 1 Crate: 0 Slot : 5 Filter 0_12

Screen Hardware

Register: Read:Write:

Read Write Write/Read

Read/Write Registers

Test Port TP0 TP1 TP2 TP3

33 5 17 255

Configuration

Refresh

Sec: Once

General

/evla/widar/stb/filter/cfg_01.xmlConfiguration File

FPGA Bit File /evla/widar/stb/filter/bit/abc Browse

Design/Revision/Version: running01 71 State:

Browse

Input delay calibration:

Status

Input delay adjust:

Delay Module Input:System clock:

Read/Write:

Reset status indicators

CRC

View Error Counts

Reset Error Counts

Input: B

InOut

AOutput Enable:

Input Delay: 37 112Daisy Delay:

Time Interval :

Delay Model: External

Delay: 123456 Delay Rate: 0

Demux Factor:

Delay0Data Input Rate: 15

Phase Factor: 0x4000

/evla/widar/stb/filter/s1prd/abcProduct File: Browse

Invalid Stretch Length :

Scale Factor : 1

Filter Delay :

Data Output Rate:

Number of Taps :

Stage 1

16

XBAR

32

/evla/widar/stb/filter/s2cof/abcCoefficient file Browse


Scale Factor : 1

Filter Delay :

Output Rate:

16

Calculation Rate:

32

Stage 2

Number of Taps



Scale Factor : 1

Filter Delay :

Output Rate:

16

Calculation Rate:

32

Stage 3

Number of Taps



Scale Factor : 1

Filter Delay :

Output Rate:

16

Calculation Rate:

32

Stage 4

Number of Taps

Format

Valids

on

313

Power off 12345

Quantizer Clip Count

Quantizer State Count

Quantizer Power

RFI Detections

TEX Valids

TEX Sums

Quantized State:

RFI Detect Level:

RFI Detect Length:

Auto -3

Select Data:

Quantizer Scaling:

Quantizer # Bits:

TEX Trig File

TEX Phase Model

TEX Phase:

TEX Phase Rate:

/evla/widar/stb/filter/fm/tex/trig/abc

External

0xFEDCBA9

0x12345678

Browse

0x17534

0x1234

4

256

Select Clip:

cos sin

on off

Output File: /evla/widar/stb/filter/fm/out/abc Browse

Clip Count

Sideband Flipper On

Histogram

VCI

Input # bands:

Output band width (MHZ):

Output band center (MHz):

Input # bits:

Input band width (MHz):

Mix:

Output # bits:

192.0000000

Input band:

Setup

/evla/widar/stb/filter/fm/out/abcResults file: Browse

Fbit:

Accumulation (interrupts): 100

Flip:

LSB:

Mixer Phase Model:

Phase Rate: 0xFEDCBA9

Phase:

Use mixer

0x00000000

/evla/widar/stb/filter/s2mix/abcMixer Trig File Browse

Mixer

External

Filter Control GUI



Correlator Software People• Sonja Vrcic (Penticton)

– Coordinates overall design and specification.– Virtual Correlator Interface (VCI) definition.– Master Correlator Control Computer (MCCC) S/W.

• Bruce Rowen (Socorro)– Correlator hardware control S/W (CMIB).

• Kevin Ryan (Socorro)– GUI development and hardware control S/W.

• Martin Pokorny (Socorro)– Correlator Backend software.

• Michael Rupen, John Romney, Bryan Butler, Ken Sowinski, Barry Clark, Bill Sahr (Socorro)

– Advisory capacity.



Correlator-related Software Issues

• Correlator output: ALMA Science Data Model (ASDM) must be qualified for EVLA use.– On the surface ASDM looks all right in the sense that additions can be

made to meet EVLA requirements.– Is ALMA effort required and is it available?– This issue is to be addressed later in the series of presentations.

• Preliminary plans to add “blocks” in the Correlator Back End (CBE).– Additional Ethernet Switch and output Fast Data Formatter

computer(s).– Formatting to ASDM spec’s will probably take place in the Fast Data

Formatter.– Details to be worked out.

• Virtual Correlator Interface (VCI) Transport protocol is to be defined.

• Potential unknown throughput issues could arise.

System Progress

• NRAO work on correlator room is progressing rapidly.• Design of overall room layout is complete.• Environmental specifications have been developed.

– Air flow & temperature: ~1700 cfm per rack, 15 oC.– ElectroStatic Discharge (ESD)

• 90 nm devices used extensively (<1/1000 thickness of human hair)• Humidity specs.• Strict handling and servicing procedures.

– Air quality• ISO 14644-1 class 8 + MERV 13 filter.

• Racks– “Thermal prototypes” have been built and tested.– “Mechanical prototypes” ditto.– Preliminary fabrication plan is being developed.

• Preliminary installation plan is being developed.• Power plant specifications/RFP draft written for

acquisition this fiscal year (ends Mar 31/07).

Correlator Rack Layout1 GbpsEthernet

to Backend

100/1000 MbpsEthernet M&C

M&CEthernetswitch

-48 VDCbreakerpanel

Up to 40, 4-wafer,1.024 Gbpscables from

station racks toFanout Boards.

-48 VDC,48VRET frompower plant

Up to 256, 4-wafer,

1.024 Gbpscables: Fanout-

to-BaselineBoards

FRONT

Raised floor

Rack pedestal

airflow

airflow

Correlator Room Layout48'-0"

Prim

Bay

47'-0

"

2-cable 48VDC connects from each DCPDU breaker output to each rack

seperately.(24 cable pairs in a 32-Station correlator)

115VAC

Fiber Fiber

HV

AC C

W U

NIT

HV

AC C

W U

NIT

SPARE - reservedHVAC 5th CW UNIT

(if req’d)

40-Station Expansionadds 2 Station racks

B

B

B

B

B

B

S40

S40

Station Racks(8)

BaselineRacks(16)

AC input for HVAC andlighting.

AC input, 50 kW, 56 kVA for backendcomputers, MCCC, CPCC, testoutlets

480 VAC, 3-phasedelta, 190 kW (211kVA) for 48 VDC plant

Sec

Bay

Battery String 1

Battery String 2

BE1

BE2

BE3

BE4

BE5

BE6

HVAC DXCOMPR UNIT

HVAC DXCOMPR UNIT

-48VDC Power Plant

Battery Strings

Backend Computers

B

B

B

B

B

B

B

B

S

S

S

S

B

B

S

S

S

S

SwitchRack

MCCC/CPCCRack

ElectroStatic Discharge (ESD)

RESTRICTED ACCESS

Overriding door interlock will soundALARM

ESD Protected Areawear ESD coats and shoes



Correlator Testing

• Production correlator chip tests– Functional acceptance test by supplier using DRAO-

supplied equipment.• At speed, but not over temperature range.

– DRAO lab tests: special jig to be fabricated.• HALT/HASS tests being investigated (advanced version of

“burn-in”).• Unit PCB hardware tests during production:

– Fast functional tests performed at factory in DRAO-supplied equipment.

• Unit hardware tests after acceptance– HALT/HASS tests of circuit boards (or equivalent).– Extended functional tests.



Correlator Testing (cont’d)

• System Tests– Prototype Stage

• Extended testing/debug cycle in lab – all functions.• Many self-tests, simulated observations.• Should confer a high degree of reliability for On-the-Sky tests.

– On-the-sky Tests• Final demonstration of prototype tests – designed to be quasi-

independent of EVLA S/W.• Begin System Integration with EVLA software (continuing

process).• Hardware left at VLA site.

– Software Tests after On-the-Sky correlator delivered.• Continued testing with hardware that will remain at the VLA.• Understand interference (RFI) and the “clues” that the WIDAR

system provides.

Test Configuration – Software View

On-the-Sky Test Setup

Fan Tray

-48 VDC Power Supply

Ethernet Switch

12Ucrate

Sta

tion

Boar

d

Base

line

Boar

d

Sta

tion

Boar

d

Sta

tion

Boar

d

Sta

tion

Boar

d Tim

ecod

e B

oard

4' 5"

27"

Host ComputerBackendComputer

To NRAO network

1 phase: 170-264 VAC (30 A)

Host ComputerHostComputer

“External Timecode” (fiber)

128 MHz CW, 0dBm

110 VAC, 15 A

From NRAOArray TimingReference

-48 VDC Breaker Panel



Installation Procedure(Note: Dates from Apr/06 Long Term Schedule)

• 06-07: Install –48V DC power system.• 06-07: Install underfloor cables.• 08: Manufacture racks in Penticton.• 08: Install overhead power distribution.• 08: Install control computers, Ethernet switches,

backend computers and other COTS equipment.• 08-09: Install boards into racks in an order that

maximizes scientific benefit.



Preliminary Installation Estimates(Note: Dates from Apr/06 Long Term Schedule)

• Q4-06/Q1-07 – power supply (-48 V DC) .– Procurement, installation, training.– 40 person-days of NRAO effort.

• Q3/Q4-07 – signal cabling.– Installation of inter-rack high-speed cabling.– 512 pre-fab cables, 3 different lengths; 24 pre-fab power monitor &

control cables.– 52 person-days of NRAO effort.

• Q4-07/Q1-08 – Racks– 24 racks, pre-fabricated, tested in Penticton.– 24 person days of NRAO effort.

• Q1-08 – power cabling.– Two cable runs: To distribution panel; distribution panel-to-racks.– 20 person-days of NRAO effort.



Preliminary Installation Estimates (cont’d)

• Q2-08 – control computers, M&C Ethernet– MCCC, CPCC, Ethernet switches, etc.– 120 VAC power required.– 8 person-days of NRAO effort.

• Q3-08 – Back-end computers and equipment.– Gbit Ethernet system.– 20 person-days of NRAO effort.

• Q3-08/Q3-09 – Board installation and test.– Occasional NRAO effort.

• Total Est. NRAO installation effort:– ≥ 8.2 Person-Months.

EVLA Correlator Group

DRAO-based.

NRAO-based.



Funding in Canada – No ChangeAug/03 – Canadian Treasury Board approval of

submitted budget ($C 20M over 5 years).

• Most spending in $US.• But see risk factors.

Correlator Projected Spending Profile

EVLA Spending Profile

$-

$1,000,000.00

$2,000,000.00

$3,000,000.00

$4,000,000.00

$5,000,000.00

$6,000,000.00

1999

/0020

00/01

2001

/0220

02/03

2003

/0420

04/05

2005

/0620

06/07

2007

/0820

08/09

2009

/10

Fiscal Year

$C

Labour

Fabrication

Contracts

Equipment

Softw are

Miscellaneous

Travel

EVLA Spending Profile

$-

$1,000,000.00

$2,000,000.00

$3,000,000.00

$4,000,000.00

$5,000,000.00

$6,000,000.00

$7,000,000.00

1999

/0020

00/01

2001

/0220

02/03

2003

/0420

04/05

2005

/0620

06/07

2007

/0820

08/09

2009

/10

Fiscal Year

$C

Labour

Fabrication

Contracts

Equipment

Softw are

Miscellaneous

Travel

Correlator Summary ScheduleVery Uncertain Time Scale

Milestone Progress



Project Management

• Schedule– Detailed schedule periodically updated.– Near-term (target) schedule updated weekly.– Long-term schedule discussed monthly, and

cross-checked against detailed schedule.• Budget

– Projections updated with schedule.• Bills of Materials (BOM’s)

– 1000’s of components.– Maintenance required.



Correlator Documentation

• Master Document Tracking Spreadsheet Maintained at DRAO.

• 115 documents written so far, including “Memos”.

• Additional 23 documents with designations and titles are anticipated.



Principal Design Reviews

• Three Design Reviews:– Conceptual (CoDR – Nov, 2001)

• Review architecture and overall design.

– Preliminary (PDR – July, 2005)• Review detailed designs before prototypes.

– Critical (CDR)• Review system before “limited production” stage.• Scheduled for Q2/Q3, 2007.



Non-Technical Program Risks• Funding

– Original allotment of funds was over five years.– NRC management was made aware in Aug/03 that this does not fit the project spending profile – they

decided that internal cash management could deal with the problem.– Will have to apply for an extension, which in the present climate presents a risk.– Worst case is that project could hypothetically be halted in April 2008.– Active cash management at HIA, NRC level.

• Schedule slippage?– Due to a slow start (already happened at the beginning).– Concerns over procurement processes.

• This risk is retired. Major procurement contracts are in place except for power supply.

– Technical progress slower than the deliberately aggressive initial schedule.– “Re-spins” will present additional schedule risk.

• Inadequate contingency?– The contingency fractions are much smaller than most high-tech projects.– Cost risk will be reduced quickly once prototypes are tested. – Exchange-rate changes have been favourable to date.

• Inflation not being recognized in funding profile?– Inflation is a corrosive influence.



Technical Program Risks• Technical risk is currently at a maximum

– Much design effort and prototype expenditure has taken place.– Prototype hardware is being fabricated (or about to be fab’d), but not

received/tested.• But …

– Considerable design effort expended to reduce technical risk.• Finely divided testing schemes for circuit boards – every path can be checked

independently.• Correlator chip underwent an independent test and verification process, including a

major simulation campaign after the “place-and-route” stage.• A separate Design for Manufacturability (DFM) analysis done for circuit boards –

results in a major reduction in risk.

• Formal system reliability analysis has been done, and will be updated.• Funds and design effort has been expended to minimize technical risk.

– We are optimistic and confident that technical risk is reasonably low.



Descoping

• The correlator is difficult to split, once designed, and saving is inefficient.

• Have not reconsidered descoping options since the budget is currently “under control”.

• If the previously-mentioned non-technical risks become imminent concerns, then de-scoping options will have to be revisited.



Project Summary• Are we meeting the required schedule?

– We are sticking with the original delivery date, although there is now some squeezing of activities towards the end.

• Are we over budget at this stage?– Budget is slimly allocated, but we are not over budget.

• Are we planning to deliver on what we said we would do?– Yes, with minor improvements.

• What are the major risks at this stage?– Hypothetical funding difficulty in 2008.– Technical risks should be reduced this year as prototypes are

tested.

the expanded very large array...5.13118 4.02882 cage1 2.125 cage2 2.775 3 4 2.450 cage5 sw1 1 2 3...

Documents