IEEE TRANSACTION ON NUCLEAR SCIENCE, JUNE 2018 1

Nanoseconds Timing System Based on IEEE 1588FPGA Implementation

D. Pedretti, M. Bellato, R. Isocrate, A. Bergnoli, R. Brugnera, D. Corti, F. Dal Corso, G. Galet, A. Garfagnini,A. Giaz, I. Lippi, F. Marini, G. Andronico, V. Antonelli, M. Baldoncini, E. Bernieri, A. Brigatti, A. Budano,

M. Buscemi, S. Bussino, R. Caruso, D. Chiesa, C. Clementi, X. F. Ding, S. Dusini, A. Fabbri, R. Ford,A. Formozov, M. Giammarchi, M. Grassi, A. Insolia, P. Lombardi, F. Mantovani, S. M. Mari, C. Martellini,

A. Martini, E. Meroni, L. Miramonti, S. Monforte, P. Montini, M. Montuschi, M. Nastasi, F. Ortica, A. Paoloni,E. Previtali, G. Ranucci, A. C. Re, B. Ricci, A. Romani, G. Salamanna, F. H. Sawy, G. Settanta, M. Sisti,

C. Sirignano, L. Stanco, V. Strati, G. Verde

On Behalf of the JUNO Collaboration

Abstract—Clock synchronization procedures are mandatory inmost physical experiments where event fragments are readoutby spatially dislocated sensors and must be glued together toreconstruct key parameters (e.g. energy, interaction vertex etc.)of the process under investigation. These distributed data readouttopologies rely on an accurate time information available at thefrontend, where raw data are acquired and tagged with a precisetimestamp prior to data buffering and central data collecting.This makes the network complexity and latency, between fron-tend and backend electronics, negligible within upper boundsimposed by the frontend data buffer capability. The proposedresearch work describes an FPGA implementation of IEEE 1588Precision Time Protocol (PTP) that exploits the CERN Timing,Trigger and Control (TTC) system as a multicast messagingphysical and data link layer. The hardware implementation

D. Pedretti, is with the Department of Information Engineering, PadovaUniversity and INFN Legnaro National Laboratories, Italy.

M. Bellato, R. Brugnera, D. Corti, F. Dal Corso, S. Dusini, G. Galet,A. Garfagnini, A. Giaz, R. Isocrate, I. Lippi, F. Marini, F. H. Sawy, C.Sirignano, L. Stanco are with the Department of Physics and Astronomy,Padova University and INFN Padova, Italy.

A. Bergnoli is with DWave.it, Padova, Italy.G. Andronico, S. Monforte, G. Verde are with INFN Catania, Italy.V. Antonelli, A. Brigatti, A. Formozov, M. Giammarchi, P. Lombardi, E.

Meroni, L. Miramonti, G. Ranucci, A. C. Re are with the Department ofPhysics and Astronomy, Milano University and INFN Milano, Italy.

M. Baldoncini, F. Mantovani, M. Montuschi, B. Ricci are with the De-partment of Physics and Earth Science, Ferrara University and INFN Ferrara,Italy.

E. Bernieri, A. Budano, S. Bussino, A. Fabbri, S. M. Mari, C. Martellini,P. Montini, G. Salamanna, G. Settanta, are with the Department of Physicsand Mathematics, Roma Tre University and INFN Roma Tre, Italy.

M. Buscemi, R. Caruso, A. Insolia are with the Department of Physics andAstronomy, Catania University and INFN Catania, Italy.

D. Chiesa, M. Nastasi, E. Previtali, M. Sisti are with the Department ofPhysics, Milano Bicocca University and INFN Milano Bicocca, Italy.

C. Clementi, F. Ortica, A. Romani are with the Department of Chemistry,Biology and Biotechnology, Perugia University and INFN Perugia, Italy.

A. Martini, A. Paoloni are with INFN Frascati National Laboratories,Frascati, Italy.

V. Strati is with the Department of Physics and Earth Science, FerraraUniversity and INFN Legnaro National Laboratories, Italy.

M. Grassi is with INFN Milano, Italy and APC Laboratory IN2P3, Paris,France.

R. Ford is with INFN Milano, Italy and SNOLAB, Ontario, Canada.X. F. Ding is with INFN Milano and Gran Sasso Science Institute, L’Aquila,

Italy.

extends the clock synchronization to the nanoseconds range,overcoming the typical accuracy limitations inferred by com-puters Ethernet based Local Area Network (LAN). Establishinga reliable communication between master and timing receivernodes is essential in a message-based synchronization system. Inthe backend electronics, the serial data streams synchronizationwith the global clock domain is guaranteed by an hardware-basedfinite state machine that scans the bit period using a variabledelay chain and finds the optimal sampling point. The validityof the proposed timing system has been proved in point-to-pointdata links as well as in star topology configurations over standardCAT-5e cables. The results achieved together with weaknesses andpossible improvements are hereby detailed.

Index Terms—timing system, synchronization, frontend elec-tronics, hardware, FPGAs, eye diagram.

I. INTRODUCTION

THE context of the proposed research work is the Jiang-men Underground Neutrino Observatory (JUNO) [1] [2].

The timing system is an essential part of this experiment,indeed, to precisely determine the energy and the interactionvertex of incident neutrinos, the charge information comingfrom the 18000 photomultipliers (PMTs) surrounding thecentral detector must be associated with a precise time infor-mation. The JUNO readout architecture foresees the frontendelectronics, hereby represented by the Global Control Unit(GCU) card, to be installed underwater, close to the PMTs[3]. The heart of the frontend electronics is an FPGA thathandles data digitization, data buffering, data readout, slowcontrol and monitoring, trigger generation and synchronizationwith the backend electronics. Trigger requests generated bydifferent channels are asynchronous and independent eventsthat must be attached with a timestamp in order to be correctlyprocessed by the central trigger system. The central triggersystem collects trigger requests coming from all the readoutchannels and generates trigger validations. The trigger valida-tion is essentially a data readout request delivered to all theGCUs with the goal of collecting data fragments in a timewindow centered around a center time parameter specified

0000–0000/00$00.00 c©2018 IEEE

arX

iv:1

806.

0458

6v2

[cs

.NI]

19

Jun

2018

IEEE TRANSACTION ON NUCLEAR SCIENCE, JUNE 2018 2

in the request message. The center time must be univocallyinterpreted by the central trigger system and by all the GCUs.This evidences how the distributed nature of the data readoutdemands for a synchronization system whose primary taskis to handle an accurate time distribution from the centraltiming and trigger system to all the frontend cards. The triggersynchronization accuracy demanded is ± 16 ns.

A. Different Approaches to Synchronization

There are essentially two different approaches to synchro-nization widely used in physics experiments: event-basedsynchronization and time-based synchronization [4]. Both arevalid solutions that emphasize different concepts of synchro-nization and both potentially lead to an accuracy in the sub-nsrange, upon precise delay and asymmetry measurement andcompensation.

1) Event-based Synchronization: The main task of anevent-based timing system is to deliver reliable, fixed andlow latency control messages to all the nodes reached bythe synchronization network. An example of event-basedsynchronization is the Micro-Research Finland (MRF) timingsystem [5]. In this topology, the event generator is the onlyholder of the global time information and it converts thescheduled timing events in optical signals delivered througha deterministic network to an array of event receivers. Eachreceiver converts the received event codes to synchronizeddigital output pulses. The automatic delay measurement andcompensation is a key feature to achieve a sub-ns resolution.

2) Time-based Synchronization: The primary task of atime-based timing system is to handle an accurate time distri-bution and clock synchronization. The precision time protocol,defined in the IEEE 1588-2008 standard, is an example oftime-based synchronization since it relies on an accurate copyof the global time held in thousands of timing receiver nodes[6]. The accuracy of a PTP software implementation over astandard Ethernet LAN rarely extends in the sub-µs range.

White Rabbit (WR) timing system is an example of time-based synchronization that exploits the IEEE 1588-2008 stan-dard and extends the timing resolution to the sub-ns rangefeaturing a 1000 base-LX Synchronous Ethernet over single-mode optical fiber and implementing a phase tracking systembased on a Digital Dual-Mixer Time Difference (DDMTD)phase detection [7].

II. THE JUNO SYNCHRONIZATION SCHEME

The choice of the timing system to be used depends onwhich services are demanded to the synchronization systemand on the data readout architecture. In JUNO, the main taskof the timing system is to generate an accurate copy of theglobal time at the backend and frontend levels of the readoutarchitecture as shown in Figure 1. The Central Trigger andTiming System (CTS) holds the global time to be distributed toall the timing receivers. The backend electronics cards (BECs)are integrated into the White Rabbit network that may providea sub-ns synchronization with the CTS. The global clocksignal must be distributed from the BECs to the frontend cardsthat count the time locally. Every local count will experience

Fig. 1. JUNO synchronization scheme overview.

an offset with respect to the global time counter since the startof the counting is not synchronized among GCUs. This offsetmust be measured and corrected.

The WR network ends at the BEC level and cannot extendto the frontend electronics since the potting of underwaterelectronics imposes tight constraints on the number of com-munication channels between the BEC and each GCU as wellas on the communication medium. The JUNO collaborationdid not envisage the use of optical fibers underwater and theadoption of WR on copper cables would nullify the benefitsof the phase tracking procedure foreseen by WR, degradingits resolution. The communication medium between BEC andGCU, on which is based this research work, is restricted to acouple of 80 m long twisted pairs in a CAT-5e cable.

The first clock alignment proposal was a synchronousreset pulse to be sent from the CTS to all the GCUs. Thissolution has been discarded since the delay calibration andcompensation over asymmetric communication media, like theCAT-5e cable, needs special hardware to be accomplished.Moreover, the synchronous reset solution is not selective and itwould preclude the possibility to execute the clock alignmentprocedure runtime. If for any reason, one channel loses thesynchronization, the operator should exclude that channel fromthe data readout until the next run of the experiment.

The offset correction mechanism between backend andfrontend electronics, object of this paper, is based on threepillars:

• an hardware implementation of the IEEE 1588-2008standard;

• clock syntonization based on the clock data recovery(CDR) strategy;

• the implementation of a full duplex and deterministiclatency communication link layer between BEC and GCUover two copper twisted pairs.

The paper details the advantages and disadvantages of the pro-posed solution supported by the experimental results achieved.

IEEE TRANSACTION ON NUCLEAR SCIENCE, JUNE 2018 3

Fig. 2. IEEE 1588 PTP offset measurement c©2008 IEEE.

III. PTP THEORY AND SYNCHRONIZATIONPERFORMANCES

A. Clock Offset Correction Mechanism

IEEE 1588-2008 standard defines a protocol for preciseclock synchronization applicable to systems that implementa multicast communication model between the master andthe timing nodes. In JUNO, assuming a messaging exchangecapability, the idea is to exploit the delay request-responsemechanism measurement defined in the IEEE 1588-2008 stan-dard to compensate for the offset error between backend andfrontend electronics. Figure 2 shows the protocol implemented.The follow-up message is not strictly necessary and is notused in the proposed solution. t1 g − t1 l is the clock offsetto be measured and compensated. The offset measurementprocedure is accomplished in eight steps:

1) The master records the current timestamp t1 g and sendsto the slave a synch messages containing the timestampt1 g .

2) The slave records the reception time t2 l . The slavecomputes: t1 g−t2 l = offset−delayms where delayms

is the transmission delay from master to slave.3) The slave sends a delay request message, without pay-

load, to the master and records the transmission timet3 l .

4) The master records the reception time t4 g .5) The master sends back a delay message containing t4 g

value.6) The slave, upon receipt of the delay resp message

computes: t4 g − t3 l = offset + delaysm .7) Now, with the assumption: delayms = delaysm , the

offset can be computed using (1):

offset =(t1 g − t2 l) + (t4 g − t3 l)

2(1)

8) the slave corrects its clock accordingly.The master individually addresses the offset correction

procedure to each slave. The procedure is periodical, thusensuring clock alignment during the run of the experimentand offering the possibility to check the synchronization statusof all the GCUs. Frequent offset corrections are indicativeof problems and the corresponding GCU should be broughtoffline for diagnosis and firmware maintenance.

B. PTP Performances and Limiting Factors

The offset correction mechanism highlights the sources oferror that potentially bound the clock alignment accuracy:

• The protocol does not specify the clock frequency; lower-frequencies lead to poorer time resolutions.

• Timestamping is a time critical operation. Hardware-assisted timestamping is required to achieve time syn-chronization in the ns range.

• The synchronization over standard Ethernet LAN rarelygoes beyond the µs of accuracy due to packet latency inthe Ethernet network that is traffic dependent. The bestperformances of PTP over Ethernet are usually achievedwith the Deterministic Ethernet, a communication tech-nology that uses time scheduling to ensure a boundedand low latency transmission of the critical scheduledmessages. The proposed timing system and the resultsdescribed in this paper rely on a deterministic latency foreach of the 18000 channels; PTP accounts for differencesin these latencies.

• Like all message-based synchronization protocols, PTPtime accuracy is degraded by asymmetry. Asymmetryusually originates from the physical medium and fromthe implementation of the data link layer. The assumptiondelayms = delaysm is not true in presence of asymmetry.Specifically, the time offset error is 1/2 of the asymmetry.

C. Advantages of a Digital Implementation

Hardware-assisted implementations of PTP over Ethernetexist and prove that performing in hardware specific tasksleads to tight time synchronization [8]. If the assumptionof a perfect symmetry holds, the theoretical resolution of afully digital implementation of PTP is ± one clock periodas shown by the temporal diagram of Figure 3. The time isimplemented in the form of a digital counter that counts theperiods of the clock signal. The local time, prior to the clockalignment procedure, differs from the global time by a randomoffset Ty −Tx , determined by the power up sequence and bythe time to lock of the phase-locked loop (PLL). The offsetcorrection mechanism may end up in a configuration in whichthe local time lags (green) or leads (blue) the global time. Bothare correct and acceptable solutions determined by the phasedifference ϕ between the global clock edge and the local clockedge; this phase difference ϕ is mainly determined by thetransmission latency. The PTP by construction cannot resolve

IEEE TRANSACTION ON NUCLEAR SCIENCE, JUNE 2018 4

Fig. 3. Offset correction temporal diagram.

Fig. 4. Positive offset correction example.

ϕ, whose measurement is usually done via phase trackingsystems (e.g. the DDMTD phase detection in WR). In theproposed timing system ϕ is unknown but in principle, withoutvariations of the cable length, it is invariant with a standarddeviation imposed by the jitter.

Two numerical examples of the offset correction are givenin Figure 4 and Figure 5. The red arrows represent thesynch and delay req messages. The timestamps are registeredon the clock rising edges that coincide with the messagestransmission and/or reception. From these examples, we candeduce that with a 250 MHz global clock frequency theexpected synchronization accuracy will be ± 4 ns.

Figure 6 shows one more example with a different ϕ thatleads to a final configuration with the local count slightlylagging the global time. The picture also highlights the serialdata synchronization issue and its impact on the time accuracy.As explained later in the paper, in the backend electronics, theinput data stream have to be synchronized with the globalclock domain. Indeed, if the data transition, indicated bythe green arrowhead, is too close to the clock rising edge,an ambiguity in the received timestamp may arise due totiming violations. t4 g might be 403, blue arrow, and thecomputed offset would be 23.5. Afterward, the division bytwo, implemented with a 1-bit right shift operation, roundsdown the offset to 23. Any asymmetry contribution less than2 clock periods does not affect the synchronization correctness.

Fig. 5. Negative offset correction example.

Fig. 6. The impact of data synchronization on the time accuracy.

IV. TIMING SYSTEM IMPLEMENTATION

A. TTC as Physical and Data Link Layer

PTP relies on a multicast communication model that ensuresbidirectional and asynchronous messaging between master andslaves. Ethernet is the mostly used interconnection model butPTP is not limited to Ethernet. The interconnection systemproposed exploits a couple of twisted pairs, in a CAT-5e cable,as a transmission medium between the master (BEC) and eachof the 48 slaves connected to it (GCUs). An overview is givenin Figure 7.

The physical and data link layers are based on the CERN’stiming, trigger and control system concept [9]. The TTCencoder and decoder implement a simple data link layer whoseprimary tasks are framing and error checking and correctionwith the Hamming codes. It does not implement data flowcontrol and handshaking mechanisms. Two communicationchannels are time division multiplexed (TDM). Channel Ais reserved for future delay calibration developments whilechannel B is used to encode broadcast commands consistingof 16-bit frames decoded by all receivers, and, individuallyaddressed commands consisting of 42-bit frames. These longframes contain a header, the receiver identification number,the receiver internal address and data fields. At the physicallayer, data is BiPhase Mark encoded (BMC) to ensure a DCbalanced transmission and a self-clocking solution.

The network and transport layers, with reference to thestandard Ethernet stack, are not provided by the TTC systemwhose aim is to implement a simple, deterministic and lowlatency bidirectional communication channel between BECand GCUs. The TTC model satisfies the requirements of the

IEEE TRANSACTION ON NUCLEAR SCIENCE, JUNE 2018 5

Fig. 7. Timing system physical and data link layers.

trigger system, timing system, and the serial link synchroniza-tion system (delay control in the figure). The trigger requestand validation messages have the highest priority since theirlatency must be bounded, with the upper bounds imposed bythe frontend data buffer capability.

B. Clock Syntonization

The JUNO timing system foresees the global clock signalto be distributed to the frontend nodes as encoded informationin the TTC messages. The local clock in any GCU refersto the syntonized copy of the global clock recovered fromthe data stream generated by the master. The syntonization isbased on a clock and data recovery Integrated Circuit (IC) thatguarantees that any local clock is locked in frequency with theglobal clock. This makes the slaves immune to medium andlong term frequency drifts that manifest as a linearly increasingphase difference and a cumulative error on the local timecount.

C. PTP Digital Design Overview

A complete overview of the digital circuit that implementsthe offset correction mechanism is given in Figure 8. Thistiming system requires the availability of two FPGAs (onein the backend card and one in the frontend card) and a fullduplex communication channel between the two. The choice ofthe communication medium bounds the maximum admissibledistances between master and slave nodes. The proposeddesign is based on a CAT-5e unshielded twisted pair (UTP)cable, therefore, the maximum distance is about 100 m withthe support of two cable driver-receiver couples. The expecteddata rate of 250 Mbps is well in the range of the high range

Fig. 8. RTL design overview.

(HR) general purpose I/O pin capability of the chosen FPGA,thus freeing the design of the communication physical layerfrom the usage of dedicated transceivers with a consequentreduction of the power consumption. The VHDL code isgeneric and might be synthesized for any FPGA manufacturerjust replacing the I/O buffers and the clock management tileswith those provided for the family chosen. The test setupimplemented exploits a Xilinx’s Kintex-7 XC7K160T but thedesign fits comfortably in a smaller size and lower powerFPGA. During the tests, the global clock signal has beenemulated with a free running oscillator. After the integrationin the WR network, it will be provided by the WR PTP core.Each frontend board recovers the global clock with the CDRand counts it locally. The CDR output data and clock buffersintroduce a fixed latency (source of asymmetry) that hasbeen measured using the Xilinx’s ChipScope debug tool andcompensated thanks to the programmable coarse delay inputstage of any TTC decoder. The offset correction protocol andmessages flow have been conceived as a couple of master-slavefinite state machines (FSMs). The CERN’s internal release ofthe TTC decoder and encoder cores have been revised andoptimized to accommodate the custom timing system require-ments. The TTC has no handshaking mechanism, therefore,the PTP master and slave cores implement a watchdog thattakes back the FSM to idle state in case that a message is notcorrectly delivered and the offset correction procedure stalls.The PTP master scheduler follows a round-robin algorithm to

IEEE TRANSACTION ON NUCLEAR SCIENCE, JUNE 2018 6

Fig. 9. Serial data streams synchronization

address the synchronization procedure to the timing receivernodes sequentially. The synchronization cycle is periodical.

V. SERIAL LINK SYNCHRONIZATION

The serial link synchronization procedure addresses themarginal capturing into the FPGAs and it is crucial to es-tablish a reliable communication channel between backendand frontend electronics [10]. As shown in Figure 9, in thefrontend electronics the CDR chip automatically locks on theinput data stream and tracks the phase of the input data inorder to shift the recovered clock to the best sampling pointminimizing the possibility of having the marginal capturingphenomena. Figure 10 shows the recovered clock togetherwith its jitter histogram and the data eye diagram. As visiblethe setup and hold times are met and the cycle-to-cycle jitterstandard deviation is 3.4 ps.

In the backend electronics, the 250 Mbps low-voltage dif-ferential signaling (LVDS) serial data streams synchronizationis more complex since 48 data streams in different phase rela-tionship must be synchronized with the global clock domain tominimize the risk of marginal capturing that may compromisethe communication stability. Techniques normally adopted tominimize the probability of metastability in digital designs,like synchronization registers, are not a feasible solution whenthe important information lies in a sequence of bits. The issuehas been addressed using a cascade of 4 programmable finedelay primitives, IDELAYE2 (ODELAYE2 primitives cannotbe cascaded because their output drives the correspondingI/O block and cannot be routed to the internal FPGA logic).Each IDELAYE2 primitive is a 31-tap wraparound selectabledelay with a calibrated tap resolution of about 78 ps. Thisfine delay block is placed at the output of the TTC encoderin any frontend board, and its tap count is remotely incre-mented/decremented from the master calibration procedurerunning in the backend FPGA. The maximum delay of thechain amounts to about 9.6 ns, enough to scan two complete bitperiods. The data stream input to the backend FPGA is delayedincrementally in steps of 78 ps and plotting the TTC frameerror count versus the tap count one can get the informationabout the eye opening and the best sampling point as illustratedin Figure 11. Running at 250 Mbps the expected data eye

Fig. 10. 250 MHz clock recovered on the GCU, CDR data output eye diagramand jitter measurement.

Fig. 11. Bathtub plot of the LVDS serial data stream capturing.

width is 4 ns that correspond to about 51 taps. The clocksynchronization procedure cannot start until this calibration iscompleted and the channel is error free. Establishing a reliablebidirectional communication between the master and all thetiming receiver nodes is essential in a 18000 channels setup.

VI. CLOCK SYNCHRONIZATION PROCEDURE ANDRESULTS ACHIEVED

The clock synchronization solution proposed has been testedand fully characterized in a test setup composed by one masterand three timing receiver nodes. In sequence, these are themain steps to get an accurate copy of the global time atfrontend level:

• power up the BEC and GCUs boards. The power upsequence of the boards is not a concern.

• The BEC card locks with the global clock signal andstarts to count the global time and broadcasts periodicalidle commands.

• Each GCU locks to the recovered global clock copy andstarts counting the time locally.

• Each GCU starts the channel identification procedurenecessary to decode the TTC commands. As soon as anidle command is correctly decoded, the channel alignedflag is set to ’1’.

IEEE TRANSACTION ON NUCLEAR SCIENCE, JUNE 2018 7

Fig. 12. Time accuracy achieved with about 3 m of CAT-5e cable.

• The BEC broadcasts to all GCUs an error reset command.Upon having decoded the error reset command, all theTTC decoders into the frontend boards are expected tobe error free.

• Enable the serial link synchronization procedure.• Check that all the communication channels are error free.• Enable the clock synchronization procedure in the master

and in all GCUs.• Once enabled, the offset correction mechanism is period-

ical.The outcome of the offset correction procedure has been

verified using an oscilloscope as an external observer. Thebackend and frontend boards have been programmed to gener-ate a pulse at a scheduled time, and, the output pulses observedwith the oscilloscope are shown in Figure 12. The pulses arealigned within 1 ns. As expected there is no control on thephase relation between the global clock signal and the localclock signals.

The test has been repeated with three cables of differentlength to reproduce a condition similar to the final installationon the field. The result is shown in Figure 13. The timeaccuracy achieved is well within the requirements of ± 16ns, but, the time offset of the GCU2 is slightly larger than the250 MHz recovered clock period. The offset error is induced

Fig. 13. Time accuracy achieved with a 3 m long cable channel 1, 80 m longcable channel 2, 50 m long cable channel 3.

by the asymmetry [11]. The only source of asymmetry in thedesign (not compensated) is the CAT-5e copper cable. Typicalpropagation delay for CAT-5e UTP is in the order of few ns permeter and the standard specifies that a 100 m cable might havea delay skew between pairs up to 50 ns, due to the differenttwist rate. Figure 14 displays the cable analysis performed onthe 80 m and the 50 m cables used to test the timing system.The asymmetry between the pairs 1,2 and 3,6 of the 80 mcable is 10 ns. This asymmetry generates the clock periodoffset error observed. Without any compensation mechanism,the offset error introduced by the cable asymmetry may be upto 25 ns. In this worst case scenario, the local clock would beout of specification.

A. Accuracy Improvements

The compensation of the asymmetrical latency introducedby the cable is necessary to claim a time resolution of ± 4 nsusing copper cables.

In a context like JUNO where cable layout cannot bechanged after installation, asymmetry could be measured at thecable supplier premises and then manually compensated usingcoarse and fine delay primitives included in the firmware.

IEEE TRANSACTION ON NUCLEAR SCIENCE, JUNE 2018 8

Fig. 14. CAT-5e UTP cables analysis.

Dedicated hardware support in frontend and backend cardscould provide the possibility of swapping transmit and receivepaths, hence allowing an automatic measurement of cablelength imbalance and consequent compensation [12].

Where applicable, the digital implementation of PTP willbenefit a lot from a TTC optical distribution. The communi-cation medium asymmetry would then be negligible obtaininga ± 4 ns timing system over an extended transmission range.

VII. CONCLUSION

A fully hardware implementation of PTP for offset measure-ment and compensation has been developed and tested. Thedigital design proposed includes the TTC system as physicaland data link layer to realize a fully duplex and deterministiclatency communication channel between master and slaves andenables the synchronization of thousands of timing receivernodes with a precision of ± one clock period. The test setupdescribed in the paper is based exclusively on the presence ofthe FPGA technology on the backend and on the frontendelectronics, with a communication medium consisting of astandard CAT-5e UTP cable. The results achieved confirm thatthe implemented timing system is a cost effective solution

to extend the time accuracy to ± 4 ns without complexcalibration procedures. The fully hardware design togetherwith the deterministic multicast communication system getsrid of the asymmetries introduced by classical PTP softwareimplementations over standard Ethernet networks with theconsequent performance improvements. The only source ofasymmetry of the proposed timing system is the physicalmedium that, if not compensated, may cause offset errors.

REFERENCES

[1] G. Ranucci, Status and prospects of the JUNO experiment. IOPConf. Series: Journal of Physics: Conf. Series 888 (2017) 012022. DOI:10.1088/1742-6596/888/1/012022.

[2] M. Grassi, Charge reconstruction in large-area photomultipli-ers. IOPscience JINST 13 P02008, 2018. DOI: 10.1088/1748-0221/13/02/P02008.

[3] D. Pedretti, The Global Control Unit for the JUNO front-end electronics.Proceedings of TIPP2017, May 22-26 2017, Beijing.

[4] A.A. Soderqvist, N. Claesson et al, Comparison of SynchronizationLayers for Design of Timing Systems. THPPC102, Proceedings ofICALEPCS 2013, San Francisco, CA, USA, ISBN 978-3-95450-139-7.

[5] (2018, June), The Micro-Research Finland Oy website. Available:http://www.mrf.fi/

[6] IEEE Instrumentation and Measurement Society, IEEE Standard for aPrecision Clock Synchronization Protocol for Networked Measurementand Control System. IEEE Std 1588

TM-2008.

DOI: 10.1109/IEEESTD.2008.4579760[7] (2018, June), The White Rabbit Project website. Available: http://white-

rabbit.web.cern.ch/[8] Texas Instruments, AN-1728 IEEE 1588 Precision Time Protocol Time

Synchronization Performance. Application Report, SNLA098A Rev.April 2013.

[9] S. Baron, Timing, Trigger and Control (TTC) systems for the LHC.Website. Available: http://ttc.web.cern.ch/TTC/intro.html

[10] Texas Instruments, design Considerations for Avoiding Timing Errorsduring High-Speed ADC, LVDS Data Interface with FPGA. Applica-tion Report SLAA592A, June 2013, Rev. May 2015.

[11] N. Simanic, Compensation of Asymmetrical Latency for EthernetClock Synchronization. 2011 IEEE ISPCS, DOI: 10.1109/IS-PCS.2011.6070151.

[12] L. Huang, Compensation for Asymmetry of Physical Line. Mar. 2011,802.1 AVB, 201103 IEEE 802 plenary.