1350 precursor talk - meerkat lessons learned (thomas abbott
TRANSCRIPT
www.ska.ac.za
SKA Engineering Meeting October 2016
PRESENTER: Thomas AbbottFROM THE INTEGRATION OF MEERKAT ARRAY RELEASE 1
LESSONS LEARNED
www.ska.ac.za
Agenda
• StoryofMeerKATAR1integration
• MeerKATAIVplan
• Tools
• LessonsLearned
www.ska.ac.za
ThestoryofMeerKATintegration
• IntegrationstartedinJune2015• AR1systemwasintegratedandworkingby~May2016• ScaledupandbugsfixedforourAR1releaseinJune2016
www.ska.ac.za
2015– Earlyintegrationishard
January2015:ReceptorTestSystemisworking.RTSisasmallcorrelator,aprevious-generationtelescopemanager,andtworeceptors.
9June: WeconnectupacompletecopyoftheMeerKATsignalpathintheintegrationlab.It’sexpectedtoworkinafewweeks.
June: LabIntegrationeffort:twice-a-weeksessionsstart
July: LabIntegrationeffort,firstfileisrecordedfromasignalgenerator
August: Labintegrationcontinues
Sept: Labintegrationcontinues.Systemnowstartsupandrunsobservations
Oct: Labintegrationcontinues.Labverificationstarts.
www.ska.ac.za
July2015
Firstfilerecordedinlab:(cross-correlationofthesignalgeneratornoiseonly)
www.ska.ac.za
Connectingreceptors
July-October2015:Back-endisinstalledonsite.
Nov2015:FirstreceptorisreleasedfromRTS.Integrationofsitecorrelatorwithreceptor.
LateDec2015:FourreceptorsconnectedtoAR1toattempttoformafirstimagebeforetheendoftheyear.Systemisnotstableenough,butweget:Firstfringes Phase closure
Jan2016:BacktotworeceptorsasRTScontinuesonthesecondpair
www.ska.ac.za
MarchandApril2016
April:Firstbeamformer observationandofflinereductionofapulsar
March:FourreceptorsconnectedFirstroughimageformed
www.ska.ac.za
20May2016
Firstimagecreatedwitha12-hourobservationon4antennas–threeinthecoreandone4kmaway.ComparisontoearliersurveysshowsthehighresolutionofMeerKAT.
20May:8-antennacorrelatorrunsforthefirsttime
www.ska.ac.za
5June2016
16-antennacorrelatorrunsforthefirsttime,11receptors
www.ska.ac.za
10June2016
10June:16th receptorconnectedacceptedandintegrated(butwehaveonefailure,soweneeda17th receptor)
The17th receptorisconnectedaweeklater
www.ska.ac.za
16June2016
Disaster!Abuginthedelaytrackingsystemsurfaces,wheresomedelaysarenotcorrectlyappliedbythecorrelator.
Saturday18June,9am,anewversionofthecorrelatorhasbeencompiledandtested.Theweekend’sobservingstarts...
www.ska.ac.za
17June2016
Successfulpulsarobservations
www.ska.ac.za
19June2016
SuccessfulcontinuumobservationsofourDEEP2fieldReducedoverthefollowingweektoformtheimage
www.ska.ac.za
20June2016
Successfulspectrallineobservationofamasercalibrator
www.ska.ac.za
16July2016
Firstlightpressrelease
www.ska.ac.za
www.ska.ac.za
Agenda
• StoryofMeerKATintegration
• MeerKATAIVplan
• Tools
• LessonsLearned
www.ska.ac.za
MeerKATIntegrationandVerification
• Arrayreleases• I&Vplan• Verification• Tools
www.ska.ac.za
Arrayreleases
DefinedintheMeerKATIntegrationandVerificationPlanStagedreleaseoffunctionalityandscale:1. Receptorconstructionschedule2. ScaleofCBFandSDP3. FunctionalityofCAM4. FunctionalityreleasesforCBFandSP
www.ska.ac.za
Arrayreleases
AR1
16receptors
Correlator4k &32k
Offlineimaging
1beam
AR2
32receptors
Correlator4k &32k
Offlineimaging
1beam
AR3
64receptors
Correlator4k &32k
Pipelineimaging
1beam
Laterreleases
64receptors
Correlator1k,4k &32k
Pipelineimaging
4beams
Narrowbandmodes
Transient modes
VLBI
www.ska.ac.za
AR1IntegrationandVerificationplan
www.ska.ac.za
AR1Verification
AR1hasthreesystem-levelverificationeventsdefined:– VE1:Labevent,testingmostbasicfunctionality– VE2:Firstsiteverification.EssentiallythesamefeaturesastestedinVE1,butnowonthesitesystemwithrealreceptorsandskysignals.
– VE3:CompleteAR1verificationonsitesystem,withatleast6receptors.
Subsystemseachqualifiedaccordingtoowndevelopmentplan
www.ska.ac.za
www.ska.ac.za
Agenda
• StoryofMeerKATAR1integration
• MeerKATAIVplan
• Tools
• LessonsLearned
www.ska.ac.za
Tools
• ReceptorTestSystem• IntegrationLabinCapeTown• ManagementTools
www.ska.ac.za
ReceptorTestSystem
Firstantennasaredeliveredandintegratedbeforethetelescopeback-endisready.
InshortyouneedaTelescopetotestaReceptorEnterReceptorTestSystems(RTS)
TotesttheemergentpropertiesofaReceptorweneed:• Controlinterface(CAM)• Networktransportofdataandcontrol
(LAN/Fibre)• Timeandfrequencyreference(TFR)• Correlator(CBF)• Signalprocessingandreduction(SDP)• SpecialisedTestreceivers(KuRX)
www.ska.ac.za
ReceptorTestSystem
www.ska.ac.za
ReceptorTestSystem
Testsystem(RTS)needsqualificationaswell• Driftscan
• KuHolography
www.ska.ac.za
ReceptorTestSystem
L-bandtesting• Singledish:
– Nominalinput(widebandgain)– Tsys andTnoise diode– Tippingcurve– Gainstability– Linearity– Spectralbaseline
• Interferometric:– Reflectorefficiencies– Phasestability– Beamshapeandsidelobe levels– Polarisation
Ku-bandtesting• Singledish
– Gaincurve– Pointing
• Interferometric– Aperturephaseefficiency– Gaincurve– Errorbeam
www.ska.ac.za
IntegrationLab
AsimpleIntegrationLaballowedearlyintegrationtobedonelongbeforethesystemonsitewasready.
Ourlabconsistedof:• 2x40Gbps inter-officeLAN• Signalgeneratorandnoisesource• SimulatedAntennaPositioner• 8-inputcorrelator:12Roaches.Correlatormastercontroller• Onedigitiserandpedestalswitch.Digitisermastercontroller• ControlandMonitoringsystem,one(virtualised)server• Scienceprocessoringestnode• RegularMeerKATsciencearchive• General-purposetimeandfrequencyreferencesystem
www.ska.ac.za
IntegrationLab
www.ska.ac.za
Managementtools
VITECHCOREformodellingofthesystem.RequirementsspecificationsandverificationplansareprinteddirectlyfromCORE
WeuseJIRAextensivelyinallteamsformanagingweek-to-weektasksandreportingbugsandfeaturerequests
Primaveraprojectplanningsoftwareforhighlevelprojectplansandreporting
www.ska.ac.za
www.ska.ac.za
Agenda
• StoryofMeerKATAR1integration
• MeerKATAIVplan
• Tools
• LessonsLearned
www.ska.ac.za
LessonsLearned
• Prototyping• Labintegration• Earlyintegration• SubsystemInvolvement• Integrationteam• Systemstability– emergentbehaviour• Real-timecommunication• Cameras• RFI
www.ska.ac.za
PrototypeEverything
“Tomeasureistoknow”-- LordKelvinIndividualsubsystemsallbuiltmockups,models,orprototypes
• Don’tbelieveeverythingyoureadonaproduct’sspecificationsheet.Testitinyoursetup!
• CBFspenttwoyearswithsevennetworkswitchvendors.Allhadabsoluteconfidenceintheirproduct,butnonemanagedtomakeitoperateasadvertised,withoutsomerework.
• Atthesystemlevel,MeerKATprototypingstartedwithXDMandKAT-7.
www.ska.ac.za
LabIntegration
Beforethesitesystemwasavailable,theLabwasusedfor:• Nearlythreemonthsofintegrationwork
betweenCAM,CBFandSPsubsystems• Characterisationofdigitiser,CorrelatorPFB
leakageproblem• Substantialdevelopmentandtestingof
correlatorgateware• Discoveredatimeoffsetbetween
polarisationsindigitiser,nowimproved.• CorrelatorF-enginenon-linearity.• Beingusedtohuntadelayed-packetbug• IntegrationtestswithPulsar-timinguser
Duringsitetesting,theLabwasusedfor:• Releasetestingofnew
CBF,SPandCAMsoftware
Ongoingapplication• Evaluationofsub-
channelPFBeffects• Integrationeffortswith
USE– PulsarTiming,MeerTRAP
• DigitiserandCorrelatorgateware testing
www.ska.ac.za
Earlyintegration
Integrationismuchharderthanexpected• Allsubsystemsperformverycareful,thoroughverificationindependently.• Still,thereareinterfacingnigglesonlyexposedduringintegration.• Besttofindtheseearlywhileitcanbecorrectedwithoutimpacting
systemscheduleorbudget.• Asdescribedearlier,ittookusseveralmonthstointegratethreesystems
andgetthemtotalkandworknormally.
Thelabenvironmentenabledthisearlyintegration.
Subsystemsshouldbringareduced-functionalityversionoftheirsystemtotheintegrationlab,6monthsbeforethefull-functionalityrelease.
www.ska.ac.za
Subsystempeopleneedtobeinvolved• Integrationcannotbedonebytheintegrationteamalone.• Initially,subsystemexpertswilldothebulkoftheintegrationwork• Later,theintegrationteamwillbeabletotakeoversomefunctions
Integrationteam
SubsystemInvolvement
Subsystemexperts
Verification
Startofintegration
time
www.ska.ac.za
StrongIntegrationTeam
Astrongintegrationteamisrequiredforthelaterstagesoftheintegrationeffort.
Thefirsthalfcanrunalmostentirelyonpeoplefromsubsystems,andasthey’reresponsiblefortheproblems,they’rethebestpeopletosolvethem.
Canbemanagedbyasmallteamof
subsystemexperts
Requiresseveralspecialistswhodoonlyintegration
Involvetelescopeoperatorsearly.Thetelescoperequiresbabysittingallnight,thetechnicalteamleadscan’tkeepthisupformonths.
Laterwhensystembehaviourbecomesthemainproblem,theintegrationteamneedsseveralindependentexpertsaswellasthesubsystemteammembers.
www.ska.ac.za
Systemstability- emergentbehaviour
Overallsystemstabilityisanelusivegoal,muchmoredifficultthangettingittoworkonce.Ittakesanobsessionforqualityinallparts,notsimplygettingittowork.Toachieveasystemthatworks95%ofthetime,everypartmustworkat99%ofthetime.Stabilityproblemsweremostlyemergentbehaviour,neweffectsthatcroppedupduringintegration,orevenonceithadbeenworkingforawhile.
www.ska.ac.za
Real-timecommunication
TheintegrationteamusedIRCtocommunicatebetweenvariousfloors,officesandsite.Italsoworkedwellonacomputerpluggedintoanyreceptor
Communicationtositewasaproblem,withtheRFIrestrictions.Nocellphones orsatellitephones.WehaveafewIPphonesinsomereceptors.
www.ska.ac.za
Camerasforsitevisibility
Thehighelevationcamerawasveryusefulforobservingthingsonsite.Withoutbeingabletocallpeoplethere,itwasgoodtobeabletowatchandsee.
SurveillancecamerasfosterasenseofclosenesstotheactionintheKaroo.Considerhavingcamerasinthecontrolroom,andmajorintegrationsites,forotherstoseewho’sthereandwhat’shappening.
www.ska.ac.za
DoNotUnderestimateRFIchallenges
Thisisanultrasonicwindspeedanddirectiontransducer.
QualifiedforuseonKAT-7andinstalledforyearsonsite.
DuringRFIqualificationoftheMeerKATWSSwediscoverthatthissensorfailsthenewlimitsby20dB
www.ska.ac.za
Summary
• StoryofMeerKATAR1integration
• MeerKATAIVplan
• Tools
• LessonsLearned
• Questions
www.ska.ac.za
ThomasAbbottMeerKATSystemsEngineer
Email:[email protected]
Contactinformation
SKASouthAfrica,aBusinessUnitoftheNationalResearchFoundation.WearebuildingtheSquareKilometreArrayradiotelescope(SKA),locatedinSouthAfricaandeightotherAfricancountries,withpartinAustralia.TheSKAwillbethelargestradiotelescopeeverbuiltandwillproducesciencethat
changesourunderstandingoftheuniverse