![Page 1: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/1.jpg)
www.ska.ac.za
SKA Engineering Meeting October 2016
PRESENTER: Thomas AbbottFROM THE INTEGRATION OF MEERKAT ARRAY RELEASE 1
LESSONS LEARNED
![Page 2: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/2.jpg)
www.ska.ac.za
Agenda
• StoryofMeerKATAR1integration
• MeerKATAIVplan
• Tools
• LessonsLearned
![Page 3: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/3.jpg)
www.ska.ac.za
ThestoryofMeerKATintegration
• IntegrationstartedinJune2015• AR1systemwasintegratedandworkingby~May2016• ScaledupandbugsfixedforourAR1releaseinJune2016
![Page 4: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/4.jpg)
www.ska.ac.za
2015– Earlyintegrationishard
January2015:ReceptorTestSystemisworking.RTSisasmallcorrelator,aprevious-generationtelescopemanager,andtworeceptors.
9June: WeconnectupacompletecopyoftheMeerKATsignalpathintheintegrationlab.It’sexpectedtoworkinafewweeks.
June: LabIntegrationeffort:twice-a-weeksessionsstart
July: LabIntegrationeffort,firstfileisrecordedfromasignalgenerator
August: Labintegrationcontinues
Sept: Labintegrationcontinues.Systemnowstartsupandrunsobservations
Oct: Labintegrationcontinues.Labverificationstarts.
![Page 5: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/5.jpg)
www.ska.ac.za
July2015
Firstfilerecordedinlab:(cross-correlationofthesignalgeneratornoiseonly)
![Page 6: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/6.jpg)
www.ska.ac.za
Connectingreceptors
July-October2015:Back-endisinstalledonsite.
Nov2015:FirstreceptorisreleasedfromRTS.Integrationofsitecorrelatorwithreceptor.
LateDec2015:FourreceptorsconnectedtoAR1toattempttoformafirstimagebeforetheendoftheyear.Systemisnotstableenough,butweget:Firstfringes Phase closure
Jan2016:BacktotworeceptorsasRTScontinuesonthesecondpair
![Page 7: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/7.jpg)
www.ska.ac.za
MarchandApril2016
April:Firstbeamformer observationandofflinereductionofapulsar
March:FourreceptorsconnectedFirstroughimageformed
![Page 8: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/8.jpg)
www.ska.ac.za
20May2016
Firstimagecreatedwitha12-hourobservationon4antennas–threeinthecoreandone4kmaway.ComparisontoearliersurveysshowsthehighresolutionofMeerKAT.
20May:8-antennacorrelatorrunsforthefirsttime
![Page 9: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/9.jpg)
www.ska.ac.za
5June2016
16-antennacorrelatorrunsforthefirsttime,11receptors
![Page 10: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/10.jpg)
www.ska.ac.za
10June2016
10June:16th receptorconnectedacceptedandintegrated(butwehaveonefailure,soweneeda17th receptor)
The17th receptorisconnectedaweeklater
![Page 11: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/11.jpg)
www.ska.ac.za
16June2016
Disaster!Abuginthedelaytrackingsystemsurfaces,wheresomedelaysarenotcorrectlyappliedbythecorrelator.
Saturday18June,9am,anewversionofthecorrelatorhasbeencompiledandtested.Theweekend’sobservingstarts...
![Page 12: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/12.jpg)
www.ska.ac.za
17June2016
Successfulpulsarobservations
![Page 13: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/13.jpg)
www.ska.ac.za
19June2016
SuccessfulcontinuumobservationsofourDEEP2fieldReducedoverthefollowingweektoformtheimage
![Page 14: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/14.jpg)
www.ska.ac.za
20June2016
Successfulspectrallineobservationofamasercalibrator
![Page 15: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/15.jpg)
www.ska.ac.za
16July2016
Firstlightpressrelease
![Page 16: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/16.jpg)
www.ska.ac.za
![Page 17: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/17.jpg)
www.ska.ac.za
Agenda
• StoryofMeerKATintegration
• MeerKATAIVplan
• Tools
• LessonsLearned
![Page 18: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/18.jpg)
www.ska.ac.za
MeerKATIntegrationandVerification
• Arrayreleases• I&Vplan• Verification• Tools
![Page 19: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/19.jpg)
www.ska.ac.za
Arrayreleases
DefinedintheMeerKATIntegrationandVerificationPlanStagedreleaseoffunctionalityandscale:1. Receptorconstructionschedule2. ScaleofCBFandSDP3. FunctionalityofCAM4. FunctionalityreleasesforCBFandSP
![Page 20: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/20.jpg)
www.ska.ac.za
Arrayreleases
AR1
16receptors
Correlator4k &32k
Offlineimaging
1beam
AR2
32receptors
Correlator4k &32k
Offlineimaging
1beam
AR3
64receptors
Correlator4k &32k
Pipelineimaging
1beam
Laterreleases
64receptors
Correlator1k,4k &32k
Pipelineimaging
4beams
Narrowbandmodes
Transient modes
VLBI
![Page 21: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/21.jpg)
www.ska.ac.za
AR1IntegrationandVerificationplan
![Page 22: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/22.jpg)
www.ska.ac.za
AR1Verification
AR1hasthreesystem-levelverificationeventsdefined:– VE1:Labevent,testingmostbasicfunctionality– VE2:Firstsiteverification.EssentiallythesamefeaturesastestedinVE1,butnowonthesitesystemwithrealreceptorsandskysignals.
– VE3:CompleteAR1verificationonsitesystem,withatleast6receptors.
Subsystemseachqualifiedaccordingtoowndevelopmentplan
![Page 23: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/23.jpg)
www.ska.ac.za
![Page 24: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/24.jpg)
www.ska.ac.za
Agenda
• StoryofMeerKATAR1integration
• MeerKATAIVplan
• Tools
• LessonsLearned
![Page 25: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/25.jpg)
www.ska.ac.za
Tools
• ReceptorTestSystem• IntegrationLabinCapeTown• ManagementTools
![Page 26: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/26.jpg)
www.ska.ac.za
ReceptorTestSystem
Firstantennasaredeliveredandintegratedbeforethetelescopeback-endisready.
InshortyouneedaTelescopetotestaReceptorEnterReceptorTestSystems(RTS)
TotesttheemergentpropertiesofaReceptorweneed:• Controlinterface(CAM)• Networktransportofdataandcontrol
(LAN/Fibre)• Timeandfrequencyreference(TFR)• Correlator(CBF)• Signalprocessingandreduction(SDP)• SpecialisedTestreceivers(KuRX)
![Page 27: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/27.jpg)
www.ska.ac.za
ReceptorTestSystem
![Page 28: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/28.jpg)
www.ska.ac.za
ReceptorTestSystem
Testsystem(RTS)needsqualificationaswell• Driftscan
• KuHolography
![Page 29: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/29.jpg)
www.ska.ac.za
ReceptorTestSystem
L-bandtesting• Singledish:
– Nominalinput(widebandgain)– Tsys andTnoise diode– Tippingcurve– Gainstability– Linearity– Spectralbaseline
• Interferometric:– Reflectorefficiencies– Phasestability– Beamshapeandsidelobe levels– Polarisation
Ku-bandtesting• Singledish
– Gaincurve– Pointing
• Interferometric– Aperturephaseefficiency– Gaincurve– Errorbeam
![Page 30: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/30.jpg)
www.ska.ac.za
IntegrationLab
AsimpleIntegrationLaballowedearlyintegrationtobedonelongbeforethesystemonsitewasready.
Ourlabconsistedof:• 2x40Gbps inter-officeLAN• Signalgeneratorandnoisesource• SimulatedAntennaPositioner• 8-inputcorrelator:12Roaches.Correlatormastercontroller• Onedigitiserandpedestalswitch.Digitisermastercontroller• ControlandMonitoringsystem,one(virtualised)server• Scienceprocessoringestnode• RegularMeerKATsciencearchive• General-purposetimeandfrequencyreferencesystem
![Page 31: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/31.jpg)
www.ska.ac.za
IntegrationLab
![Page 32: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/32.jpg)
www.ska.ac.za
Managementtools
VITECHCOREformodellingofthesystem.RequirementsspecificationsandverificationplansareprinteddirectlyfromCORE
WeuseJIRAextensivelyinallteamsformanagingweek-to-weektasksandreportingbugsandfeaturerequests
Primaveraprojectplanningsoftwareforhighlevelprojectplansandreporting
![Page 33: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/33.jpg)
www.ska.ac.za
![Page 34: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/34.jpg)
www.ska.ac.za
Agenda
• StoryofMeerKATAR1integration
• MeerKATAIVplan
• Tools
• LessonsLearned
![Page 35: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/35.jpg)
www.ska.ac.za
LessonsLearned
• Prototyping• Labintegration• Earlyintegration• SubsystemInvolvement• Integrationteam• Systemstability– emergentbehaviour• Real-timecommunication• Cameras• RFI
![Page 36: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/36.jpg)
www.ska.ac.za
PrototypeEverything
“Tomeasureistoknow”-- LordKelvinIndividualsubsystemsallbuiltmockups,models,orprototypes
• Don’tbelieveeverythingyoureadonaproduct’sspecificationsheet.Testitinyoursetup!
• CBFspenttwoyearswithsevennetworkswitchvendors.Allhadabsoluteconfidenceintheirproduct,butnonemanagedtomakeitoperateasadvertised,withoutsomerework.
• Atthesystemlevel,MeerKATprototypingstartedwithXDMandKAT-7.
![Page 37: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/37.jpg)
www.ska.ac.za
LabIntegration
Beforethesitesystemwasavailable,theLabwasusedfor:• Nearlythreemonthsofintegrationwork
betweenCAM,CBFandSPsubsystems• Characterisationofdigitiser,CorrelatorPFB
leakageproblem• Substantialdevelopmentandtestingof
correlatorgateware• Discoveredatimeoffsetbetween
polarisationsindigitiser,nowimproved.• CorrelatorF-enginenon-linearity.• Beingusedtohuntadelayed-packetbug• IntegrationtestswithPulsar-timinguser
Duringsitetesting,theLabwasusedfor:• Releasetestingofnew
CBF,SPandCAMsoftware
Ongoingapplication• Evaluationofsub-
channelPFBeffects• Integrationeffortswith
USE– PulsarTiming,MeerTRAP
• DigitiserandCorrelatorgateware testing
![Page 38: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/38.jpg)
www.ska.ac.za
Earlyintegration
Integrationismuchharderthanexpected• Allsubsystemsperformverycareful,thoroughverificationindependently.• Still,thereareinterfacingnigglesonlyexposedduringintegration.• Besttofindtheseearlywhileitcanbecorrectedwithoutimpacting
systemscheduleorbudget.• Asdescribedearlier,ittookusseveralmonthstointegratethreesystems
andgetthemtotalkandworknormally.
Thelabenvironmentenabledthisearlyintegration.
Subsystemsshouldbringareduced-functionalityversionoftheirsystemtotheintegrationlab,6monthsbeforethefull-functionalityrelease.
![Page 39: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/39.jpg)
www.ska.ac.za
Subsystempeopleneedtobeinvolved• Integrationcannotbedonebytheintegrationteamalone.• Initially,subsystemexpertswilldothebulkoftheintegrationwork• Later,theintegrationteamwillbeabletotakeoversomefunctions
Integrationteam
SubsystemInvolvement
Subsystemexperts
Verification
Startofintegration
time
![Page 40: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/40.jpg)
www.ska.ac.za
StrongIntegrationTeam
Astrongintegrationteamisrequiredforthelaterstagesoftheintegrationeffort.
Thefirsthalfcanrunalmostentirelyonpeoplefromsubsystems,andasthey’reresponsiblefortheproblems,they’rethebestpeopletosolvethem.
Canbemanagedbyasmallteamof
subsystemexperts
Requiresseveralspecialistswhodoonlyintegration
Involvetelescopeoperatorsearly.Thetelescoperequiresbabysittingallnight,thetechnicalteamleadscan’tkeepthisupformonths.
Laterwhensystembehaviourbecomesthemainproblem,theintegrationteamneedsseveralindependentexpertsaswellasthesubsystemteammembers.
![Page 41: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/41.jpg)
www.ska.ac.za
Systemstability- emergentbehaviour
Overallsystemstabilityisanelusivegoal,muchmoredifficultthangettingittoworkonce.Ittakesanobsessionforqualityinallparts,notsimplygettingittowork.Toachieveasystemthatworks95%ofthetime,everypartmustworkat99%ofthetime.Stabilityproblemsweremostlyemergentbehaviour,neweffectsthatcroppedupduringintegration,orevenonceithadbeenworkingforawhile.
![Page 42: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/42.jpg)
www.ska.ac.za
Real-timecommunication
TheintegrationteamusedIRCtocommunicatebetweenvariousfloors,officesandsite.Italsoworkedwellonacomputerpluggedintoanyreceptor
Communicationtositewasaproblem,withtheRFIrestrictions.Nocellphones orsatellitephones.WehaveafewIPphonesinsomereceptors.
![Page 43: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/43.jpg)
www.ska.ac.za
Camerasforsitevisibility
Thehighelevationcamerawasveryusefulforobservingthingsonsite.Withoutbeingabletocallpeoplethere,itwasgoodtobeabletowatchandsee.
SurveillancecamerasfosterasenseofclosenesstotheactionintheKaroo.Considerhavingcamerasinthecontrolroom,andmajorintegrationsites,forotherstoseewho’sthereandwhat’shappening.
![Page 44: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/44.jpg)
www.ska.ac.za
DoNotUnderestimateRFIchallenges
Thisisanultrasonicwindspeedanddirectiontransducer.
QualifiedforuseonKAT-7andinstalledforyearsonsite.
DuringRFIqualificationoftheMeerKATWSSwediscoverthatthissensorfailsthenewlimitsby20dB
![Page 45: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/45.jpg)
www.ska.ac.za
Summary
• StoryofMeerKATAR1integration
• MeerKATAIVplan
• Tools
• LessonsLearned
• Questions
![Page 46: 1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott](https://reader031.vdocuments.us/reader031/viewer/2022022418/58a027d51a28ab444a8c4ef1/html5/thumbnails/46.jpg)
www.ska.ac.za
ThomasAbbottMeerKATSystemsEngineer
Email:[email protected]
Contactinformation
SKASouthAfrica,aBusinessUnitoftheNationalResearchFoundation.WearebuildingtheSquareKilometreArrayradiotelescope(SKA),locatedinSouthAfricaandeightotherAfricancountries,withpartinAustralia.TheSKAwillbethelargestradiotelescopeeverbuiltandwillproducesciencethat
changesourunderstandingoftheuniverse