evn vlbi [email protected] tow may 2017 · n17c1 jive 6cm nme 512 0.69 l+r 6cm nme + ftp-ft. cl17c1...

EVN e-VLBI TOW May [email protected]


Intarwebs


Harro Verkouter


e-transfer?

Harro Verkouter


Real-time VLBI!

• anything/anyone **** up?

• anything/anyone not keeping up?

you lose science data!



Why on earth e-VLBI?


‘Old style’ VLBI ...

Dec Jan Dec Jan



Dec Jan Dec Jan

∆t ≫several months!



1. Address time scaleget data to scientist faster



Dec Jan Dec Jan



Dec Jan Dec Jan

OHNOES!




2. Reliabilitydetect + fix errors before science run


Sensitivity


Sensitivity ∼ 1 ∕σT


Tsys

√(∆ν ⋅ ∆t)σT =

σT noise∆ν bandwidth

∆t integration time



Tsys

√(∆ν ⋅ ∆t)σT =

σT noise∆ν bandwidth

∆t integration time


∆ν☝︎ ⇒σT☟



Recorded data rate ∆ Sensitivity (wrt 1Gbps)1 Gbps + 0%2 Gbps + 41.4213562...%4 Gbps + 100%



`whole’ EVN equipped with• DBBC2• FiLa10G

⇒ 4Gbps capable


F17L1 JIVE 18cm FTP-FT 512 0.23 L+R 18cm FTP Fringe TestEP103A Paragi FRB121102 1024 5.99 L+R 18cmCL17L1 Gunn 18cm FS CAL ---- 0.00 L+R 18cm Amplitude Calibration EM127A Maan B0525+21 1024 3.26 L+R 21cmEM127B Maan B2045-16 1024 2.53 L+R 21cmEK036C Kirsten Crab Nebula 1024 0.92 L+R 18cm 3rd epoch N17L1 JIVE 18cm NME 512 0.69 L+R 18cm NME + FTP-FTGV022A Varenius Arp 220 2048 11.06 L+R 18cm +VLBA+Y27EP102 Paragi J0959+1259 1024 4.61 L+R 18cm +eMERLINN17C1 JIVE 6cm NME 512 0.69 L+R 6cm NME + FTP-FTCL17C1 Gunn 6cm FS CAL ---- 0.00 L+R 6cm Amplitude CalibrationEP103B Paragi FRB121102 2048 11.06 L+R -EL056 Landoni SDSS 0040-0915 2048 5.53 L+R -N17P1 JIVE 90cm NME 256 0.35 L+R 90cm NME + FTP-FTCL17P1 Gunn 90cm FS CAL ---- 0.00 L+R 90cm Amplitude CalibrationGK049C Kirsten B1133+16 256 2.88 L+R +GBT+Y27 B1133+16GK049D Kirsten B1737+13 256 2.88 L+R +GBT+Y27 B1737+13F17K1 JIVE 1.3cm FTP-FT 1024 0.46 L+R 1.3cm FTP Fringe TestGG081C Gomez AGN Jets 1024 6.91 L+R 1.3cm EVN+RA OJ287EC057B Cutini S51803+78 128 0.35 L+R 2nd epochN17K1 JIVE 1.3cm NME 1024 0.69 L+R 1.3cm NME + FTP-FTGG081D Gomez AGN Jets 1024 0.46 L+R +RA 0823+033GG081E Gomez AGN Jets 1024 0.69 L+R +RA OJ287GG081F Gomez AGN Jets 1024 0.69 L+R +RA+VLBA (HnScNl) OJ287GG081G Gomez AGN Jets 1024 0.46 L+R VLBA ONLY+RA 0823+033 EB060A Bach Cygnus A 1024 4.61 L+R -GG081H Gomez AGN Jets 1024 0.69 L+R VLBA ONLY +RA OJ287CL17K1 Gunn 1.3cm FS CAL ---- 0.00 L+R 1.3cm Amplitude CalibrationGG081I Gomez AGN Jets 1024 0.69 L+R +RA OJ287GG081J Gomez AGN Jets 1024 0.69 L+R +RA+VLBA (HnScNl) OJ287F17Q1 JIVE 0.7cm FTP-FT 1024 0.46 L+R 0.7cm FTP Fringe TestGK050 Koyama Mrk 501 1024 6.91 L+R 0.7cm +VLBAEP101A Pilhstrom Q-band Calibrators 1024 0.46 L+R 0.7cm field 1EP101B Pilhstrom Q-band Calibrators 1024 0.92 L+R 0.7cm field 2EP101C Pilhstrom Q-band Calibrators 1024 0.46 L+R 0.7cm field 1EP101D Pilhstrom Q-band Calibrators 1024 0.92 L+R 0.7cm field 2GG081L Gomez AGN Jets 1024 0.69 L+R 1.3cm VLBA ONLY +RA OJ287GA032D Agudo NRAO 150 1024 5.33 L+R 0.7cm +VLBAEP101E Pilhstrom Q-band Calibrators 1024 0.46 L+R 0.7cm field 1EP101F Pilhstrom Q-band Calibrators 1024 0.92 L+R 0.7cm field 2N17Q1 JIVE 0.7cm NME 1024 1.38 L+R 0.7cm NME + FTP-FTCL17Q1 Gunn 0.7cm FS CAL ---- 0.00 L+R 0.7cm Amplitude CalibrationEP101G Pilhstrom Q-band Calibrators 1024 0.46 L+R 0.7cm field 1EP101H Pilhstrom Q-band Calibrators 1024 0.92 L+R 0.7cm field 2

EV

N S

ession I/2017


EVN Sensitivity

> 1Gbps observations sparsesolely

due to recording media shortage!





3. Sensitivity/capacitynot limited by recording media any more





3. Sensitivity/capacitynot limited by recording media any more

4. FUN!!playing with novel technologies!


CORIntarwebs


Intarwebs


Thus: e-VLBI necessities1. get your `telescope’ connected

• fix “last mile problem”• 2017: sometimes still prohibitively expensive!

• may need sucking up to Research Network Provider• some charge commercial rates:

• ~ few 1000 currency units / time unit / bandwidth• 2017: sometimes still prohibitively expensive!

• sometimes even political issues i.s.o. money

2. switch to Mark5 recorder or newer

Any of these fail, you can forget it


CORIntarwebs


Simplified network diagram!


Intarwebs


control s/w

Mark5 system

mem

Anatomy of Mark5

PCI bus

I/O board

StreamstorCard

bank a bank b


control s/w

Mark5 system

PCI bus

I/O board

StreamstorCard

bank a bank b

disk

in

net

file

record = on / off


control s/w

Mark5 system

mem

in2net = on / off

PCI bus

I/O board

StreamstorCard

bank a bank b

disk

in

net

file


• TCP• TCP BIC• TCP Reno• ...

Started off with TCP

Long fat pipes: large BW x delay producte.g. RTT Netherlands - Australia ~ 270ms⇒ unfavourable for TCP


“let’s try udp”

we were in for a few surprises ...

• connections were dedicated light paths• did not have to be very polite to other users• we can tolerate loss of ~ few % easily

• our data is pure noise!


Current:

for( pkt=0; pkt<npkt; pkt++) {write(socket, data, 1500);

}

The n00b does UDP ...


Current:

• packets don’t even leave the Mark5! • udp has no rate control/feedback (handled by O/S for tcp)

• kernel: no obligation to actually deliver the packet(s) ...• sender overwrites own send buffer ...• have to invent ingestion rate control

The n00b does UDP ...


δt

Use Linux nanosleep() system call:

for( pkt=0; pkt<npkt; pkt++) {write(socket, data, 1500);nanosleep( delta_t );

}

For 1Gbps/MTU1500 delta_t(δt) is ~12 microseconds.Linux not real-time O/S:

threads only scheduled every 10ms (100Hz)!

n00b reads manpage


δt (~10ms)!

Use Linux nanosleep() system call:

for( pkt=0; pkt<npkt; pkt++) {write(socket, data, 1500);nanosleep( delta_t );

}

n00b reads manpage


δt

Use (pseudo) calibrated busy-loop:

int N = delta_t * counts_per_microsecond;

for( pkt=0; pkt<npkt; pkt++) {write(socket, data, 1500);for(int dummy=0; dummy<N; dummy++);

}

n00b thinks he smart


δt (random)

Use (pseudo) calibrated busy-loop:

int N = delta_t * counts_per_microsecond;

for( pkt=0; pkt<npkt; pkt++) {write(socket, data, 1500);for(int dummy=0; dummy<N; dummy++);

}

n00b thinks he smart


δt

Use wallclock time; busy wait until time for next packet:

for( pkt=0; pkt<npkt; pkt++) {// remember time of sendingtime_sent = gettimeofday();// send packetwrite(socket, data, 1500);// wait for wallclock time to pass ...time_next = time_sent + delta_t;while( gettimeofday() < time_next ) {}

}

n00b don’t give up


δt

Use wallclock time; busy wait until time for next packet:

for( pkt=0; pkt<npkt; pkt++) {// remember time of sendingtime_sent = gettimeofday();// send packetwrite(socket, data, 1500);// wait for wallclock time to pass ...time_next = time_sent + delta_t;while( gettimeofday() < time_next ) {}

}

n00b now know how!


0 1 2 3 4 5 6 97 8

So far only sender side!

UDP = no guarantees:packet losspacket reordering

Sender sends in sequence:

4 1 3 5 6 7 9 138 12Receiver gets this:


0 1 2 3 4 5 6 97 8

Put back in order

4 1 3 5 6 7 9 138 12

Sender sends in sequence:

Receiver gets this:

receiver must do this in real-time

FF


Put back in order• 3 generations of algorithm, up to 2048Mbps

• initializes buffer by overwriting (memcpy)• lots of statistics:

• reordering amount (RFC4737-ish)• loss statistics etc.

• 4th generation good up to 8192Mbps test• do not initialize• flag positions with data• only overwrite locations w/o data

• memory touched exactly once• still all statistics


Intarwebs


CORIntarwebs


bsNeed Nsta ⋅ x Gbps


bs1 x 40Gbps Haystack


bs3 x 10Gbps JIVE


bs Again: “last mile problem”In 2017:still not solved ∀correlators


bsCOR


bs

FPGA: delay, cornerturn

Custom chip: correlate


bs

Custom chip: correlateextremely synchronous ...• no buffering• no handling of network jitter


bs

FPGA: delay, cornerturnextremely synchronous ...• lost sync if too few valid time stamps found

• e.g. due to packet loss ...• correlation breaks badly!

• also no buffering


bs

CPU: delay, cornerturn, correlate


bs

CPU: delay, cornerturn, correlate extremely asynchronous!• no guaranteed throughput, may not keep up• need ~ few 100 compute nodes for 16 Gbps

• CPUs inefficient for 2-bit data


bs

CPU: delay, cornerturn, correlate • per packet overhead is a concern• takes to handle bad/missing data

• node waiting for data that will never arrive• look valid but is invalid: hard to detect!


bs

CPU: delay, cornerturn, correlate • JIVE’s SFXC software correlator works!• Cluster internal network (architecture and

throughput) is important• Investe in enough compute nodes


Something completely

different


CORIntarwebs

Data flow


Correlator Control GUI


CORIntarwebs

Control flow!


Who does what in e-VLBIField System Correlator control

time sync formatter (which channels)

frequency setup start/stop transfer

antenna (re)configure network

recorder (re)configure destination

...

Correlator must have direct access to sender!


JIVEThe NetYour station

ccs.jive.eu

fs.yourstation.tld

TCP/IP controlFirewall issues


Monitoring

Monitoring

Monitoring

Monitoring

Monitoring


Real-time VLBI!

• anything/anyone **** up?

• anything/anyone not keeping up?




if( problem )figure_out_why( time(NULL)* );

time(NULL) = ‘now’


http://services.jive.nl/sfxc/fringe.html

Real-time fringeplot on the web


Daunting?Yes.

Real-time VLBI requires:a lot of knowledgeabout a lot of subjectsby a lot of peoplesupported by a lot of tools


Real-time VLBI in EVN

Organized as monthly 24hr sessionsFuture e-VLBI Observing Sessions (real-time correlation)

-------------------------------------------------------2016 Dec 06 - Dec 07 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 Jan 17 - Jan 18 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 Feb 14 - Feb 15 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 Mar 23 - Mar 24 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 Apr 11 - Apr 12 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 May 09 - May 10 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 Jun 20 - Jun 21 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm

http://www.evlbi.org/evlbi/e-vlbi_status.html

ToO ~ scheduled on few days notice


4hr of test/setup

1. clocksearch2. possible station changes

new network h/wnetwork setup, IPv4uses diff. equip from scheduled

3. AOB


Danke schön!

evn vlbi [email protected] tow may 2017 · n17c1 jive 6cm nme 512 0.69 l+r 6cm nme + ftp-ft. cl17c1...

Documents