Nuclear Instruments and Methods in Physics Research A293 (1990) 125-129North-Holland

Section III. Project overviews: non-accelerators

INITIAL OPERATION A

CURRENT STATUS OF THE FE

LAHARDWARE CONTROL AND MONITOR SYSTEM

Robert GOODWIN, Robert FLORIAN, Marvin JOHNSON, Alan JONES and Mike SHEAFermilab *, Batavia, IL 60510, USA

DO is a large colliding-beam detector at Fermilab. The control system for this detector includes 25 VMEbus-based 68020computers interconnected using the IEEE-802.5 token-ring local-area network. In operation, the system will monitor about 15000analogue channels and several thousand digital status bits, interfaced to the 68020 computers by the MIL/STD-1553B multiplexeddata bus. In addition, the VME control system uses a memory-mapped multi-VMEbus interconnect to download parameters to morethan 100 VME data crates in the experiment. Remote host computers can then read and set memory in the detector crates over thenetwork by accessing memory in the control crates. This is an extremely useful feature during the construction phase, becauselow-level diagnostics and testing of all the detector electronics can be done over the token-ring network using either IBM-PCcompatible computers or the laboratory-wide VAX system. The VME control-system hardware is now being installed in the DOmoveable counting house. Installation is expected to be complete later this year.

1. Introduction

Fermilab is constructing a large colliding-beam de-tector for use at the DO straight section of the Tevatron .This paper describes the VME-based hardware controland monitoring system for the DO detector, with par-ticular emphasis on recent changes that have occurred,and the use of this systern during the construction andinstallation period. An earlier description of this systemis given in ref. [1].

2 . Overview

The DO detector includes over 100000 electronicchannels with equipment housed in 72 relay racks onthe detector platform and 75 relay racks in a three-storymoveable counting house . The complexity of detectorsof this size requires a control and monitoring systemsimilar to those used for a medium-sized particle accel-erator.

Fig . 1 is ra diagram of the DO detector showing theinterconnection of the control system and the detector

A. :- A_a-__ 6uoc.e lives ev~torr+c eraCIMLIULELUS. i'Vr LILID UGLGLbLV1, uaar.7~+ anv a~vwaaao auv

separate . A group of eight unidirectional 32-bit datacables transport digitized detector information to theVAX computers for processing. The unidirectional fea-ture simplifies the design of the fast data acquisition,but it requires another path to the digitizer electronicsfor downloading pedestal information and operating

*

Operated by Universities Research Association, Inc. undercontract with the US Department of Energy.

parameters. For the DO detector, the second path isprovided by the control system.

Both the control system and the digitizer electronicsuse VME. Although the VME standard specifies 3Uand 6U crates, the economics of building large quanti-ties of identical circuits favors the use of large formatcards. Detector electronics are built on 9U x 280 mmand 9U x 400 mm cards. The control system crates are6U x 160 mm. Except for the minor annoyance of pro-viding card extenders for use with standard 6U x 160mm cards, the oversized card has not been a problem.The selection of VMEbus. as a well-documented digitalstandard to follow, has been a major benefit as it allowsthe use of commercial VME cards and unambiguouslydefines the back-plane interface for all the custom cardsdesigned for DO.

3. Architecture

0168-9002/90/$03 .50 © 1990 - Elsevier Science Publishers B.V . (North-Holland)

-EASE

125

The DO control system includes a distributed groupof 68020-based local stations housed in VMEcrates andnetworked together using an IEEE-802.5 token ring.

This part of the project, known as the hardwarecontrol and monitoring system, is the subject of the

present paper (see the unshaded portion of fig . 1) . Some

data sources within the project, such as gas systems, the

argon refrigerator and the first-level trigger, are inter-

faced through IBM-compatible or VAX computers.

These systems are not treated here.There are three areas in the DO facility : the detector

itself and the platform that supports it, the moveable

counting house and the fixed counting house. The plat-

III . NON-ACCELERATORS

126

PiGat"A gä

ardware

R. Goodwin et al. / The Fermilab DO VME-based hardware control and monitor system

Fixed Counting House

I

Moveable Counting House

TokenRing .MultistationAccess Unit

Global Token Ring to CentralLaboratory Area

form houses the analogue processing electronics . Themoveable counting house contains the digitizers, thefirst-level trigger electronics and the control-system lo-cal-station computers . Host computers and the micro-VAX farm that forms the second-level trigger are locatedin the fixed counting house.

Analogue and digital monitor data are transportedto the local stations using the MIL-1553B multiplexeddata bus hereafter referred to as the 1553 . This standardwas chosen to minimize cabling required for monitorsignals and also to reduce the amount of digital noiseon the platform. The 1553 data bus uses a transformer-coupled shielded twisted-pair cable and has no continu-ously running clock, a feature that makes the cabledormant except when data is being transferred .

Since the previous description of the system [1),several changes have occurred . Local stations have beenexpanded to accommodate more analogue and digitalchannels, the CPU card was changed from a 68000 cardto a 68020, a serial vertical interconnect (see section 4.3)

Fig . 1 . System architecture for the DO detector .

replaced the original parallel version, and geographichardware identification. registers have been added (seesection 4.6).

4.1. Local stations

Flagorna

I

TokenRing MultistationI Access Unitfor Installationandstartup

b

ô

Â

Q

e`'o

ôô

Each local station is a stand-alone microprocessor-based system that includes a VME crate, a CPU card, anonvolatile mernory, an interface to the network, a localdatabase for the equipment it controls, support for asmall local console and the I/0 cards needed to con-nect to the controlled. equipment .

There are about 25 local stations in the DO controlsystem : si;c for reading the 200 muon detector monitorboards, six for montoring the 72 racks of equipment onthe plaform, four for monitoring racks and download-ing parameters to digital crates on the second and thirdfloor, of the counting house, six for downloading to thehigh-vantage system on the first floor and a few miscel-taneous systems that perform tasks such as readingcryostat temperatures and monitoring argon purity .

Local stations repetitively read analogue values fromthe equipment at a 15 Hz rate and make the resultsavailable to any requester over the network . Repetitive

R. Goodwin et al. / The Fermilab DO VME-based hardware control and monitor system

A31

A24 A2.3

AIS

and one-shot requests for data are supported. Becauseeach station has a local database containing the names,calibration factors and the nominal and tolerance valuesof its parameters, it can scan the data it reads to checkfor analog or digital alarm conditions . Alarms are re-ported on the network and can be processed by anyother computer on the network. It is expected thatportions of the local database will be downloaded to thelocal stations from the master hardware database main-tained in the host computers .

4.2. Data acquisition

Monitor data is read into the local stations using the1553 multiplexed data bus . A VME dual 1553 controllerconnects the local station to the data-acquisition hard-ware. For interfacing to equipment in a single relayrack ., a rack monitor chassis was designed as a 1553remote terminal . This chassis includes 64 ADC inputchannels, 4 words of digital I/O and 8 DAC outputchannels. This self-powered 1U chassis is placed innearly every relay rack in the counting house and on theplatform . Local stations read eight rack monitors oneach of the the two 1553 controller channels for a totalof 1024 analogue input channels per local station .

Each of the 200 muon chambers includes a custommonitor board that is similar to a rack monitor. Themuon. monitor board contains 32 analogue input chan-nels, 2 words of digital I/O and 4 DAC output chan-nels . Each of six local stations reads a maximum of 12muon monitor boards on each of the four cables drivenby two 1553 controller cards .

Including both the rack monitors and the muonsystem, the installed capacity of 16000 channels ofanalogue data and 2000 bytes of digital data can becolle~~ted by the local stations in about 18 ms.

4.3. Vertical interconnect

The token-ring-local-station data path is used fordownloading, parameters to the detector digital datacrates . : .6t. serial vertical interconnect was developed tomap 16 Mbyv° blocks of control crate memory into each

i - Memory and I/U Space

Fig. 2. Memory space of vertical-intercdnnect crates.

of four digital data crates as shown in fig. 2 . The upperbyte of the 32-bit local station address selects the slavecrate and the lower three bytes form the address for thememory access cycle in the slave crate . In the localstation, these transfers simply appear as slow memorycycles, so there is no setup or resource reservationrequired. Access to the bus in both crates is gainedthrough the normal VMEbus arbitration . Byte, word,long-word, standard 24-bit addressing and 16-bit shortI/O addressing transfers are supported transparently .The vertiaA interconnect uses AMD TAXI serial trans-mitttirs and receivers operating at a serial bit rate of 60MHz. A long-word transfer occurs in about 4 ~Ls. Asmany as 24 slave crates are served from six vertical-in-terconnect master cards in a single local station .

4.4. Network

4.5. High voltage

A8 Al

Master Crate

Slave Crate

4.6. Geographic identification

127

The control-system network is a commercially avail-able IEEE-802 .5 token-ring system operating at 4Mb:t/s . Because the alarm checking is performed bythe local stations, only the monitor data requested byconsoles or other nodes will appear on the network .This traffic is expected to be small . Although there areseveral hundred kbytes of data to download, this data isnormally sent at the beginning of each run.

A VME-based high-voltage system is under develop-ment for DO. Each high-voltage crate contains six 8-channel cards along with a CPU card that reads andcontrols the supplies . One loci station is ue,acatcca toeach of six racks of high-voltage equipment . Parametersfor the high-voltage supplies are read and set using thevertical interconnect .

In a very large repetitive system, a single wiring errorcould cause one group of electronics to inadvertentlyreceive data intended for a different group . For exam-ple, if two vertical-interconnect cables were switched,

III . NON-ACCELERATORS

128

5. Software

5.1 . Cyclic activities

5.2 . Asynchronous activities

R Goodwin et al. / The Fermilab DO VME-basedhardware control and monitor system

the pedestals for one crate would be sent to anothercrate. To help avoid such problems, we are including aregister on most of the equipment that can be readalong with the normal monitor data. For the rack andmuon monitors, an ID register is included as a separate1553 subaddress. For identification of VME digitalcrates, a small board was designed to plug into the rearof the P1 connector. This board has a single read-onlyregister accessed in the short I/O space of the crate.Higher-level software can read these ID registers toverify that they are located at the expected address.

The local-station software has been in use in theFermilab linac for 7 years. A more recent version isused for the Loma Linda medical accelerator controls.Since the previous paper on the plans for the system'sadaptation for DO [1], some significant changes havebeen made . A decision to upgrade from using an 8 MHz68000 CPU to a 20 MHz 68020 CPU was justified onthe basis that fewer local stations are required, as eachcan handle more channels. The addition of the pSOSmultitasking operating-system kernel provides the sys-tem with standardized support for task scheduling,memory allocation, event handling and message queu-ing, without loss of execution efficiency.

The system software runs many of its tasks cycli-cally . Each 15 Hz cycle, synchronized with an interruptsignal, the update task reads the raw data into a datapool of analog and binary readings, fulfills repetitivedata requests from the network and returns the answerto the requester . The alarm task scans all analogue andbinary data for alarm conditions, encoding any result-ing alarm messages for delivery to the network. Theconsole task checks and updates local-console hardwarein response to keyboard, knob or pushbutton activity.The current application-page program runs, using thefresh data from the data pool and/or that received fromother nodes. Later in the cycle, the server task fulfillsdata server requests and returns the answers to therequester .

The network task processes messages that have beenreceived from the network. The serial task processeslines of serial input, received under interrupt control atrates of up to 19200 baud, to handle real-time clocktime-of-day messages, or to save for use in satisfyingdata requests for serial data .

5.3. On-demand applications

A recent addition to the system software is on-de-mand applications . A host computer can send a mes-sage that results in a selected application being invokedon a local station, whether or not a local console isattached . The host program prepares parameters for useby the application and monitors its progress, presentingdiagnostic information to the user at the host displayterminal and collecting the resulting data. This featureis used to measure liquid-argon temperatures and argonpurity in the DO detector, where many local 1553hardware accesses are required to gather the raw dataneeded to compute these values so that the measure-ment must be done only occasionally and at timeswhich do not interrupt physics data-taking. Details ofthe implementation written to support this feature froma VAX are available in ref. [2].

5.4. Functional group addressing

To make alarm messages available to multiple nodes,group functional addressing is used on the token-ringnetwork to provide a "multicast" form of addressing. Inthis way, the messages only have to be transmitted once,independent of the number of nodes that receive them .This same scheme may be used for data server requests .

5.5. Network layer

The network layer supports general task-to-taskcommunication across the network, using a messageheader designed for use in Fermilab accelerator controls[3]. It will be used to support the network messageprotocol that has been designed for general use withinDO.

5.6. Data server

The data server support allows a host to make a datarequest containing references to multiple nodes andsend it to one of the nodes. That node uses the networkto request answer fragments from the various contrib-uting nodes, collect these fragments and rearrange themfor delivery to the original requesting node . This cansimnlifv data reauest handlino by the host .s ~

a

v r

5.7. Remote CRT image

Many local stations in DO will not have a localconsole. In the Fermiiab system, the operator interac-tion is carried out by means of "pages" . Although mostapplication pages. prw;de en»a_t access to the data fromaiiy local station, the CRT image application page canbe used to access a station remotely and invoke applica-

5.8. Data-access table customization

6. Use in system-level testing

R Goodwin et al. / The Fermilab DO VME-basedhardware control and monitor system

tions that need to be run locally, such as the 1553 testpage .

Identical software is used in each local station . Thecustomization of each station comes via informationstored in nonvolatile system tables. Chief among thesetables is the data-access table, which prescribes how andunder what conditions the data pool is to be filled foreach 15 Hz cycle. Nearly all of the control data col-lected is read from 1553 hardware, most of which isaccessed in blocks of 32 words from rack monitors.Multiple controllers using "message complete" inter-rupts allow overlapped 1553 data collection . As a result,using two controllers, 1024 channels of 1553 analogdata can be read in 18 ms. Without overlap, this takes35 ms.

The control system is expected to be essential fordetector commissioning . The digitized data is bufferedlocally on each ADC board and transferred to a bufferin the data cable driver card in the local crate as soon asall digitizers in that crate have completed their oper-ations . During normal operation, the buffer in the datacable driver would be switched from the input (VME)side to the output (data cable) side as soon as thetransfer across the crate back-plane was finished andthe data sent to the level-2 processors. A diagnosticmode inhibits this transfer so that all the data from thecards and the buffer can be read out by any VMEmaster in the crate . This can be a local personal com-puter or the control system . There is additional hard-ware which allows the trigger system to be bypassed, sothat a crate can be run in nearly a stand-alone mode,thus allowing nearly complete debugging of each crate,independent of nearly the whole normal data-acquisi-tion system and much of the control system . All that isrequired is either a local computer or the operation ofthe token ring between the test computer and the crate .This has already been used extensively for test-programdevelopment and checking . Much of the work is nowdone from the programmer's office rather than drivingto the exneriment . In the future, we hope that hardware

7. Current status

Acknowledgements

References

129

problems will also be diagnosed by experts from theiroffice . Ref. [4j describes the diagnostic system in moredetail .

Most of the local stations are now installed andoperating in the moveable counting house. A few verti-cal interconnects are being used for testing the flash-ADC and calorimeter systems . Cables for the 1553connection to the platform and muon systems have onlyrecently been delivered, but some rack monitors andmuon chambers are being operated using temporarycables.

The schedule calls for full-scale cosmic-ray testing ofall detector systems except the end calorimeters duringthe summer of 1990, and the first beam test is expectedin May 1991 .

The system described here is the result of the com-bined efforts of many people. In particular, David Kew-ley and Jerry Blazey implemented the on-demand appli-cation page that measures liquid-argon temperaturesand argon purity . Al Frank and Rich Mahler are re-sponsible for the rack-monitor and the muon-monitormodules . Jim Engelbrecht fabricated the local consoles .Many colleagues and DO collaborators have contributedto the design and implementation of various parts of themonitor system .

[1] M.F. Shea, R.W. Goodwin, M.E. Johnson, R.J . Florian andA.A. Jones . Third Int. Workshop on Accelerator Controls,Villars sur Ollon, Switzerland, 1987, CERN Yellow Report,to be published .

[2j DO Note #878 (Fermilab, 1989).[3) C . Briegel, these proceedings (Int . Conf . on Accelerator and

Large Experimental Physics Control Systems, Vancouver,BC, Canada, 1989) Nucl. Instr. and Meth . A293 (1990) 235 .

[4] R.D . Angstadt, F.O . Borcherding, G . Dougherty, M . Fort-ner, M.E . Johnson, J . Kotcher, I.L . Manning and D.C.Voy, ibid ., p . 130 .

III . NON-ACCELERATORS