apecs 2.0 upgrading the apex control system
DESCRIPTION
Dirk Muders, Heiko Hafok, MPIfR, Bonn. APECS 2.0 Upgrading the APEX Control System. APECS History APECS 2.0 Development The Upgrade Jan. 2009 Changes compared to APECS 1.1. APECS = A tacama P athfinder E xperiment C ontrol S ystem. APECS Origins. - PowerPoint PPT PresentationTRANSCRIPT
APECS 2.0 Training, APEX / ESO, Jan. '09 1
APECS 2.0APECS 2.0Upgrading the APEX Control SystemUpgrading the APEX Control System
Dirk Muders, Heiko Hafok, MPIfR, BonnDirk Muders, Heiko Hafok, MPIfR, Bonn
APECS History APECS 2.0
Development The Upgrade
Jan. 2009 Changes
compared to APECS 1.1
AP
EC
S =
Ata
cam
a P
athf
inde
r E
xper
imen
t Con
trol
Sys
tem
APECS 2.0 Training, APEX / ESO, Jan. '09 2
APECS Origins
ALMA Test Interferometer Control Software (TICS) was immediately usable at APEX due to common hardware interface (CAN bus to Vertex ACU/PTC computers)
Decided to re-use ALMA Common Software (ACS) and TICS and to benefit from large development team (a dozen people at the time in the year 2000)
Hoped for astronomer interfaces, but things developed differently...
APECS 2.0 Training, APEX / ESO, Jan. '09 3
TICS
TICS provides CORBA access to the antenna via ACS components to set up observing patterns
These components run under VxWorks to fulfill the real-time CAN bus protocol
Testing the ALMA prototypes was performed mainly via low-level Python scripts using those components directly
A higher-level user interface was planned, but it was not implemented during the test activities
APECS 2.0 Training, APEX / ESO, Jan. '09 4
What else was needed for APE(X)CS ?
To use ACS and TICS at APEX all our devices (instruments, auxiliary hardware, etc.) needed to be represented as CORBA components
An astronomer-friendly user interface to set up typical sub-mm observing scans
An online calibration pipeline was required to provide data products for astronomers
APECS 2.0 Training, APEX / ESO, Jan. '09 5
APECS Component Interface
In contrast to ALMA, APEX was always supposed to have many different instruments (receivers and spectrometers), facility and PI
Thus we decided to define generic high-level instrument interfaces to be used by all devices of the same kind
This facilitates the setup of observations enormously
APECS 2.0 Training, APEX / ESO, Jan. '09 6
SCPI Interface Level
The low-level hardware control systems could not use CORBA directly (ACS footprint was too large; CORBA programming is complex (even in ACS))
Use SCPI (Standard Commands for Programmable Instrumentation) ASCII protocol via sockets to communicate between CORBA components and hardware controllers
SCPI command hierarchy is equivalent to ACS component name space (e.g. APEX:HET345:LO1:MULTI2:biasMode)
APECS 2.0 Training, APEX / ESO, Jan. '09 7
SCPI Communication
Decoupling of CORBA and hardware controllers proved to be extremely useful because it isolates APECS from hardware which is
developed by many different groups it allows to plug in simple emulator scripts to
simulate a full system of instruments for APECS developments without real hardware
all CORBA components can be created fully automatically without any further manual interaction
the employed UDP socket protocol allows to debug hardware problems in many different ways
ACS Container
APECS SCPI Setup
Real
Hardware
Simulated
Hardware
(emuEmbSys)
CORBA
Component
ACS
ORB
CORBA
SCPI
SCPI
APECS 2.0 Training, APEX / ESO, Jan. '09 9
Multi-Beam FITS (MBFITS)
The lack of a good format to store array receiver data of single-dish radio telescopes led to the development of the MBFITS raw data format
MBFITS stores the instrument and telescope data in a hierarchy of FITS files
MBFITS is now being used at APEX, Effelsberg and the IRAM 30m telescopes
APECS 2.0 Training, APEX / ESO, Jan. '09 10
APECS Design
APECS is designed as a pipeline system starting with a scan description (“scan object”) and eventually leading to data products
The pipeline is coordinated by the central “Observing Engine”
Most APECS applications are written in Python, but they use a lot of compiled libraries to speed up computations and transactions
The astronomer interface is an IPython shell
APECS 2.0 Training, APEX / ESO, Jan. '09 11
APECS' Astronomer CLI
APECS 2.0 Training, APEX / ESO, Jan. '09 12
The APECS Pipeline
For each scan the Observing Engine accepts a Scan Object from a user CLI and Sets up the receivers (tuning; amplifier calibration) Configures the backends Sets up auxiliary devices such as synthesizers, IF
processors or the wobbler Tells the telescope to move in the requested pattern Starts and stops the data acquisition Asks the Calibrator to process the raw data to
produce the final data product or result
APECS 2.0 Training, APEX / ESO, Jan. '09 13
The APECS Pipeline (ctd.)
All communications between the Observing Engine and devices or other high-level applications is done via ACS/CORBA
APECS 2.0 Training, APEX / ESO, Jan. '09 14
APECS 0.1
The first version of APECS was installed in Chile in September 2003
Much of the time was spent on hardware installations (network, racks, servers)
We had to fight initial problems with failing pressurized hard disk boxes and missing infrastructure
APECS 2.0 Training, APEX / ESO, Jan. '09 15
Initial APECS Installation (09/'03)
APECS 2.0 Training, APEX / ESO, Jan. '09 16
Initial APECS Installation (09/'03)
APECS 2.0 Training, APEX / ESO, Jan. '09 17
APECS 0.2-1.1
Following the initial phase, APECS was debugged and extended a lot during the antenna and early instrument commissioning
We continued to deliver patches and new releases to provide control for more complex instruments and observing modes and to fix bugs
APECS 1.1 is still based on ACS 2.0.1 and runs on machines with RedHat Linux 7.2
APECS 2.0 Training, APEX / ESO, Jan. '09 18
The APECS Upgrade
Why upgrade at all ? Mainly the old Linux is making trouble Data rates ever increase and new servers are
needed to handle them RH 7.2 can no longer be installed on them In addition, ACS 2.0.1 has a number of known
issues that have been cured in later versions Old Linux libraries limit the development of new
APECS applications
APECS 2.0 Training, APEX / ESO, Jan. '09 19
APECS 2.0 Development
The upgrade plan already came up in 2004 In an effort led by J. Ibsen, ALMA had ported
TICS to ACS 3.0/4.0 The plan was to stay with VxWorks since the
ALMA RT Linux developments were still unstable
The ongoing APEX commissioning delayed our APECS porting
Only in 2007 we found the time to port the APECS and TICS codes to then ACS 5.0
APECS 2.0 Training, APEX / ESO, Jan. '09 20
The VxWorks Drama
Unfortunately, we discovered bugs in the ACS property monitoring that required ACS 6.0.4 or higher to be used to avoid container crashes
But that ACS version no longer worked with VxWorks since the ACE/TAO libraries had become to big for the PowerPC architecture
A joint effort of ESO, UTFSM, the Keck Observatory, Remedy IT in the Netherlands and many hours of Bogdan's time reanimated VxWorks under then ACS 7.0.2 again in 2008
APECS 2.0 Training, APEX / ESO, Jan. '09 21
APECS 2.0 Software
APECS 2.0 is based on ACS 8.0 and Scientific Linux 5.2 (this is ahead of ALMA who use SL 4.4 !)
Using an up-to-date ACS allows to benefit again from future ALMA developments and bug fixes
Most of the TICS and APECS functionality was kept aligned to APECS 1.1
Some areas are more advanced in APECS 2.0 (e.g. Calibrator → Heiko’s presentation)
APECS 2.0 Training, APEX / ESO, Jan. '09 22
APECS 2.0 Software (ctd.)
Basic operator / observer interfaces unchanged Many improvements such as:
Jlog with automatic filter loading New Qt GUI behavior (e.g. Calibrator Client) New Qt widgets available New Python (2.5) New libraries (e.g. SciPy, NumPy)
New Gildas (12/2008) which fixes weighting errors when combining spectra
APECS 2.0 Training, APEX / ESO, Jan. '09 23
APECS 2.0 HardwareName Type CPU RAM Disks Location
control2 HP ProLiant DL380 G4
Dual Xeon HT 3.2 GHz
4 GB 2 x High-Altitude SCA Disks (RAID 1)
New server room
instruments2 HP ProLiant DL380 G4
Dual Xeon HT 3.2 GHz
4 GB 2 x High-Altitude SCA Disks (RAID 1)
Instrument container
apexdev2 HP ProLiant DL380 G4
Dual Xeon HT 3.2 GHz
4 GB 2 x High-Altitude SCA Disks (RAID 1)
New server room
display2 HP ProLiant DL360 G5
Dual QuadCore Xeon E5345
4 GB 1 x High-Altitude SCA Disk, DotHill RAID
New server room
opt2 Industry PC Core2Duo T2500 4 GB 1 x High-Altitude SCA Disk
Instrument container
APECS 2.0 Training, APEX / ESO, Jan. '09 24
APECS 2.0 Deployment
Control2
CORBA Services
Observing Engine
Display2
FitsWriter
Calibrator
Instruments2
CORBA Containers with Instr. Comp.
SCPI Parsers
Opt2
Frame Grabber Comp.
DB2 Data Base
Apexdev2
Observing Clients
Monitoring Clients
Instrum
ents
Abm
CORBA Container with Antenna Components
An
ten
na
(AC
U, P
TC
, WIU
)CORB
A
SCPI
CAN
Bus
APECS 2.0 Training, APEX / ESO, Jan. '09 25
Software Changes [1]
Names of some variables have changed: APEXROOT → APECSROOT APEXCONFIG → APECSCONFIG APEXSYSLOGS → APECSSYSLOGS
CORBA component name separator “/” instead of “:”, e.g. “APEX/RADIOMETER/RESULTS”
Component access (needs above syntax): apexObsUtils.sc.get_device →
apexObsUtils.sc.getComponentNonSticky
APECS 2.0 Training, APEX / ESO, Jan. '09 26
Software Changes [2]
Property access compatible with APECS 1.1: apexObsUtils.getMCPoint with arbitrary separator
chosen from “/”, “:” and “.”, e.g. apexObsUtils.getMCPoint(‘APEX:HET345.state’) or apexObsUtils.getMCPoint(‘APEX.HET345/state’)
List of components: apexObsUtils.sc.COBs_available →
apexObsUtils.sc.availableComponents Enums in MonitorQuery are now strings instead
of integers (e.g. SHUTTER_OPEN)
APECS 2.0 Training, APEX / ESO, Jan. '09 27
Software Changes [3]
Web scripts now need to run on “opt2” to fetch data from the DB2 (e.g. for the weather page)
Python wrapper for Gnuplot has been removed from SciPy. Use Pylab / matplotlib instead.
Observer account administration now on “apexdev2”
APECS 2.0 Training, APEX / ESO, Jan. '09 28
Operational Changes [1]
Separate AMD maps to access /apexdata in Sequitor (lastarria:/apex_archive) and Chajnantor (display2:/apexdata) to minimize microwave link traffic. Initial reduction on “apexdev2”. Later maybe also on “display2”.
System VNC: control2:1 Observing VNC(s): apexdev2:1 Other VNC(s) (Wobb. GUI, SHFI, etc.):
apexdev2:2/3/4 Observing accounts only on “apexdev2”
APECS 2.0 Training, APEX / ESO, Jan. '09 29
Operational Changes [2]
Jlog now automatically loads the right filter (“jlog_apecs”); just need to sort by time stamps
Syslogs are now split per hour to reduce link traffic
Syslogs can be loaded into “jlog” for inspection (“jlog –u $APECSSYSLOGS/APECS-<time stamp>.syslog.xml”)
APECS 2.0 Training, APEX / ESO, Jan. '09 30
Operational Changes [3]
Two new accounts: “apexops”: pointing & focus model, system source
& line catalog administration in $APECSCONFIG “apexdata”: raw & science data and obslog areas
are owned by this account to avoid manipulations via the “apex” account whose password is not secret. The “apexdata” password must not be given to observers !!
APECS 2.0 Training, APEX / ESO, Jan. '09 31
Operational Changes [4]
As a consequence the data production programs have to run under “apexdata”
This is accomplished by special scripts that use corresponding “ssh” and “sudo” commands: fitsWriter start | stop | restart onlineCalibrator start | stop | restart obsLoggerServer start | stop | restart obsEngine start | stop | restart (for convenience)
Do not use the old restart scripts ! The overall system still runs under “apex” !!
APECS 2.0 Training, APEX / ESO, Jan. '09 32
Operational Changes [5]
tycho, apexOptRun and MonitorQuery now to be run on “opt2”
DB2 entries now obey the complete naming hierarchy
“stopAllAPECSClients” now stops observer processes too. Always use “restartAPECSServers” which includes stopping the clients. Restart takes only 6 min. now !
ABM console now via “minicom 1” on “opt2”
APECS 2.0 Training, APEX / ESO, Jan. '09 33
Operational Changes [6]
Offline reductions at the MPIfR are now performed on “display2” (in Bonn) under the “apex” account. The data areas are mounted as /aux/apexa and /aux/apexb.
APECS 2.0 Training, APEX / ESO, Jan. '09 34
New Network Setup
In addition to the new APECS hardware, the network has been upgraded to Gbit (reached 55 MB/s (!) for file transfers) and physically split into these subnets at Chajnantor: Control (CORBA, SCPI): 10.0.2.* Data (TCP streams fromt backends to FitsWriter on
“display2-d”): 10.0.5.* (not routed !) Maintenance (everything else: web cams, thin
clients, notebooks, etc.) 10.0.6.* Additional “SciOps” subnet for observational
machines in Sequitor: 10.0.7.* (pular, etc.)
opt 2Optical CameraTerminal to ABM
instruments2instrument control
control2ObsEngine
CORBA-ServicesVNCServer
display2FitsWriterCalibrator
apexdev2Development
APECSvncserver
Hardware devices
Pointing Telescope
Backends
Telescope/WobblerABM
Monitoring Things WEB Cams tcserver
pularVNC client
apexdevVNC client
tacoraVNC client
apexdb2VNC client
VNC Connections
SequitorChajnantor
Control netw
ork, Data netw
ork
Maintenance network
lastarriaArchive
rsync
APECS 2.0 Training, APEX / ESO, Jan. '09 36
APECS 2.0 Testing [1]
During the installation a number of tests have been made to verify the performance: Initial drive tests: no strange motions / vibrations Tracking tests at different az/el: like in APECS 1.1 Optical pointing runs: agree with previous results Radio scans with SHFI and SZ (calibration,
pointing, focus, skydip, on, raster, otf (also holo mode), spiral raster) produced MBFITS and Class data as expected.
APECS 2.0 Training, APEX / ESO, Jan. '09 37
APECS 2.0 Testing [2]
Throughput tests: AFFTS with 28 x 8192 channels @ 10 Hz (8.75
MB/s !) without delays or other problems SZ beam maps (which failed in Nov. 2008) with 141
subscans, OTF, continuous data (i.e. one 25 minute subscan !) without problems
PBEs and SZ backend are already in the data network. Spectrometers will be changed by B. Klein during his next visit
APECS 2.0 Training, APEX / ESO, Jan. '09 38
Issues
APECS 2.0 is good, but not perfect: ABM NC tasks (az/el & wobbler pos.) crashes
sometimes. Being iterated with B. Jeram (ACS). optView GUI freezes after “movie” mode Gildas as of 12/2008 has some bugs. Will need to
update to a more recent version soon. Cron jobs handling syslogs need to be adjusted.
APECS 2.0 Training, APEX / ESO, Jan. '09 39
APEX Staff To Do List
All Linux machines in Sequitor should be updated to SL 5.2
At least “llaima” should be available for offline data reduction with SL 5.2 and APECS 2.0 soon
The LDAP and DNS services need to be moved to new servers (ideally off of normal PCs)
The remaining web cams etc. need to be reconfigured to the maintenance subnet
Port “fieldtalk” programs for optical camera to SL 5.2
APECS 2.0 Training, APEX / ESO, Jan. '09 40
Conclusions
APECS 2.0 with new network performs much better than the previous system
We are no longer bound to very old hardware and can fulfill requirements of new instruments
Observing modes that started failing last year are now possible again
New observing modes with higher data rates can now be used
Future developments facilitated by new Linux / libraries.