next gen. correlator workshop, groningen, june 2006, a. szomoru, jive new use for an old correlator...
TRANSCRIPT
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
NEW USE for An old correlatoR
Arpad Szomoru
Joint Institute for VLBI in Europe
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Mark IV EVN Correlator
• Developed by international consortium: EVN institutes, MIT
• Officially inaugurated October 1998• Comparable correlators in use at
Haystack Obs., US Naval Obs., MPIfR, JIVE
• Correlator boards in use at WSRT, SMA
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Basic specifications
Input:• No. of telescopes (N) 16• Array Observing frequencies: 329 MHz – 22 GHz• Data bandwidth: 128 MHz per polarization (Right and
Left-hand circular), divided into 8 bands. • Channel bandwidths 0.5 MHz to 16 MHz.• Data input rate: 1 Gbps per telescope
Output:• Integration time 1/4s (will become 1/32s with PCInt)• 2048 spectral channels per baseline/band/polarization • Data Output rate 6 MB/s (will become 80 MB/s with PCInt)
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Field of View limitation
• Limited by tint
• Time smearing
• Shorter integrations• Enable wide field
surveys• Study μJy sources• Discriminate AGNs
• But, enormous increase of output data volume..
VLBI FoV x 100
6 arcmin [FWHP] Effelsberg beam
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
EVN MkIV Correlator limits• Integration time
• Cycle time of 0.015s (actually, 1/64th of a second)
• Spectral resolution• 131072 complex lags per readout = 65536 spectral points
per readout• Divided over 32 products leads to 2048 spectral channels
per product
PCINT :• Short for Post Correlator Integrator• Capture the full output of the EVN MkIV
correlator to disk• Need to replace output datapath
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
The PCInt project
• High speed readout of the correlator was already prepared• Via DSP powered serial port
• Need hardware and software to enable this• Receiving end of serial port• Gbit ethernet for transfer from correlator rack to data collection host• Fast disk subsystem in order to support 160MByte/s (parallel RAID
arrays in a Storage Area Network)
Harro Verkouter
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Curre
nt situ
atio
n
CCC
EEE (xk)
DDD (xn)
Switch
FC Switch
Correlator Board (x8)
RaidArray (xm)
RT System
Ethernet Card
C40 COMM
VME High Speed Serial
SBC (x2)
Correlator rack (x4)
PCI
100TX1000FX(x8)
Fibre Channel
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Phase
0
CCC
EEE (xk)
DDD (xn)
Switch
FC Switch
Correlator Board (x8)
RaidArray (xm)
RT System
VME High Speed Serial
Correlator rack (x4)
100TX1000FX(x8)
Fibre Channel
Ethernet Card
C40 COMM
SBC (x2)
PCI
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Phase
1
CCC
EEE (xk)
DDD (xn)
Switch
FC Switch
Correlator Board (x8)
RaidArray (xm)
RT System
VME High Speed Serial
Correlator rack (x4)
100TX1000FX(x8)
Fibre Channel
Ethernet Card
C40 COMM
SBC (x2)
PCI
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Phase
2
CCC
EEE (xk)
DDD (xn)
Switch
Correlator Board (x8)
RaidArray (xn)
RT System
VME High Speed Serial
Correlator rack (x4)
100TX1000FX(x8)
1000TX
Ethernet Card
C40 COMM
SBC (x2)
PCI
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Post-processing issues• Huge data volumes
1hour @160MByte/s equals560GBytes of data
• Use a cluster of nodes
• Need automated processing• Use a processing pipeline
Achieved 1/16s sampling, at 24 MB/s data outputUsers seem to be ready for 800 GB
data-sets…
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Data Acquisition
•Move from tape to disk recording
•Reliability•Cost•Bandwidth•Efficiency
•e-VLBI: the next step•No consumables•Higher bandwidth•Fast turn-around•ToO support
Disk based recording
e-VLBI using fiber
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Why e-VLBI ?
• Reliability – real-time feedback to the telescopes• Logistics – No media management• Sensitivity – sustained data rates >> 1 Gbps possible…
• Rapid science results:
• Geodesy (Earth rotation rate)• Precision spacecraft navigation• Transient phenomena… GRBs, SNe etc.
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Why e-VLBI (cont) ? • Target of Opportunity (ToO) capability:
• Dominated by VLBA currently…
• Reliability & Logistics e-VLBI
• Sensitivity e-VLBI
• Rapid science e-VLBI
• Rapid publication e-VLBI
• Optimal observing strategy (obs. freq., calibrators, telescope array)
• LOFAR Transients etc. ToOs may become much more common e-VLBI.
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
e-VLBI Proof-of-Concept Project
• DANTE/GÉANT Pan-European Network• SURFnet Dutch NREN• GARR Italian NREN • UKERNA UK NREN• PSNC Polish NREN • DFN German NREN• KTHNOC/NORDUnet Nordic NREN • Manchester University Network application software • JIVE EVN Correlator• Westerbork telescope Netherlands• Onsala Space Observatory Sweden• MRO Finland• MPIfR Germany• Jodrell Bank UK• TCfA Poland • CNR IRA Italy
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
0SISISISI
GRGRGRGR
CZCZCZCZ
IEIEIEIE
LULULULU
EEEEEEEESKSKSKSK
ILILILILLTLTLTLTCYCYCYCY
ESESESES
HUHUHUHUITITITIT
FRFRFRFR
UKUKUKUK
BEBEBEBE
NLNLNLNLDEDEDEDE SESESESE CHCHCHCH
PTPTPTPT
ATATATAT
GEANTGEANTGEANTGEANT
2.5 G 1.2 G 622M
155 M
BGBGBGBGLVLVLVLVRORORORO
HRHRHRHR
PLPLPLPL
45 M34 M
310 M
GÉANT:Access of NRENs to GÉANT
EVN telescope
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
POC results
• Demonstration of feasibility• Identification of problems• Has led to closer ties with networking community and
generated political interest• Has laid the foundation for the next step forward
(EXPReS):
• I3 proposal to the EC (Communication & Network Development Call)
• Ranked first out of 43 proposals; nearly fully funded to an amount of 3.9 MEuro.
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
EXPReS – major aims:
• Making e-VLBI an operational astronomical instrument:
• 16 telescopes connected to JIVE at 1 Gbps • Robust real-time e-VLBI operations• Transparent inclusion of e-MERLIN antennas within e-EVN• Target of Opportunity Capability
• Future developments in e-VLBI
• >> 1 Gbps data transfer rates, • extended LOFAR etc. • distributed software correlation.
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Network testing
•Use existing protocols on currently available hardware
•TCP maximal reliability•Not really required
• Sensitive to congestion•Lot of fine-tuning necessary•And possible
•UDP connectionless• Unaccountable
•Tailor made protocols?•Lambda switching
•Internet weather•Hard to quantify•Hard to pinpoint bottlenecks
0
100
200
300
400
Dw- BoBo- DwDw- Bo*Bo- Dw*
0
200
400
600
800
Dw- Bo
Dw- Tr
Dw- OnM
bps
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
• February 2005: network transfer test (BWCTL) employing various network monitoring tools involving Jb, Cm, On, Tr, Bologna and JIVE
Network testing (2)
Onsala-Jive
0
200000000
400000000
600000000
800000000
1000000000
1200000000
1 3 5 7 9 11
13
15
17
19
21
23
25
TCP
UDP
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
First real-time eVLBI Image, 3 telescope observation of gravitational lens, May 2004
First eVLBI science observation, OH masers around IRC10420, Richards et al, Oct 2004
First broadband eVLBI science, detection of theHypernova SN2001em, Paragi et al, astro-ph/0505468
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
• First open e-EVN Call for Proposals (March 2006)
• First Target of Opportunity Observations (May 2006, Cygnus X-3), Tudose et al. (in prep)…
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Current status
• Regular science runs at 128 Mbps with 6 European stations (24 hours)
• Arecibo sometimes participates at 32 Mbps
• Fringes from all European stations at 256 Mbps have been demonstrated, and,
• on single baselines, 512 Mbps
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Issues, Developments
Convincing a correlator designed for tape technology to become real-time..
Operational improvements:• Robustness• Reliability• Speed• Ease• Bandwidth• Station feedback
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
OngoingNew control computers (Solaris AMD servers)• Cut down on (re-)start time• Powerful code development platform• Tightening up of existing code
Other hardware upgrades:• Upgrade existing connectivity from 6*1 Gbps to
16*1 Gbps (lightpaths)• SX optics (fibres + NICs)• Replacement of SU functionality: Mark5A→B:
motherboards, memory, power supplies, serial links.
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Conclusions
• e-VLBI is changing the nature of VLBI• Fast response, ToO capability• better quality control, rapid data delivery• New science, higher bandwidths
• Large fields of view• Will allow the study of μJy sources• Or many masers over a large star formation region• Data archive will contain millions of weak sources
• EXPReS will realize an operational e-VLBI network distributed across 1000’s km – a true pathfinder for SKA.