IN DEGREE PROJECT TECHNOLOGY,FIRST CYCLE, 15 CREDITS

, STOCKHOLM SWEDEN 2017

Practical analysis of thePrecision Time Protocol underdifferent types of system load

EMIL GEDDA

ANDERS ERIKSSON

KTH ROYAL INSTITUTE OF TECHNOLOGYSCHOOL OF COMPUTER SCIENCE AND COMMUNICATION

Practical analysis of thePrecision Time Protocol underdifferent types of system load

EMIL GEDDA

ANDERS ERIKSSON

Bachelor Thesis in Computer ScienceDate: June 7, 2017Supervisor: Pawel HermanExaminer: Örjan EkebergSwedish title: Praktisk analys av IEEE 1588 under olika typer avsystembelastningSchool of Computer Science and Communication

i

AbstractThe existence of distributed real-time systems calls for protocols for highaccuracy time synchronization between devices. One such protocol, thePrecision Time Protocol (PTP) reaches sub microsecond synchronizationprecision.

PTP can be implemented both in hardware and software. This studyaimed to analyze how system stress could affect the accuracy and preci-sion of software implemented PTP between two devices. This was doneusing two Intel Galileo Generation 2 running Linux systems. Softwarewas used to simulate CPU, I/O, network, and OS usage. Data was ex-tracted from software logs and summarized in charts and then analyzed.

The results showed that PTP synchronization accuracy and precisiondoes suffer under certain types of system load, most notably under heavyI/O load. However the results might not be applicable to real-worldscenario due to limitations in hardware and the synthetic stress testsdo not correspond to real-world usage. Further research is required toanalyze why and how different types of system load affects PTPs accuracyand precision.

ii

SammanfattningFörekomsten av distribuerade realtidssystem kräver protokoll för nog-grann tidssynkronisering mellan enheter. Ett sådant protokoll, PrecisionTime Protocol (PTP), kan uppnå en precision på under mikrosekundenunder synkronisering.

PTP kan implementeras i både hårdvara och mjukvara. Den här rap-porten fokuserar på att analysera hur systembelastning kan påverka pre-cision och noggrannheten hos mjukvaruimplementerad PTP mellan tvåenheter. Testen utfördes på två stycken Intel Galileo Generation 2 kö-randes Linux. Mjukvara användes sedan för att simulera belastning påolika system såsom CPU, I/O, nätverk och på operativsystemet. Dataextraherades ifrån loggar från mjukvaran, vilken sammanfattades i graferför att sedan analyseras.

Resultaten visade att precisionen och noggrannheten hos PTP för-sämras under vissa typer av systembelastningar, mest märkbart undertung I/O belastning. Resultaten är dock potentiellt inte applicerbara påverklighetscenarion på grund av begränsingar i hårdvaran samt att syn-tetiska stresstest inte motsvarar normal belastning. Ytterligare forskningkrävs för att analysera hur och varför olika typer av systembelastningpåverkar PTPs precision och noggrannhet.

Contents

1 Introduction 11.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . 2

1.2.1 Research question . . . . . . . . . . . . . . . . . . 21.3 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Background 42.1 The Precision Time Protocol . . . . . . . . . . . . . . . . 4

2.1.1 Best master clock algorithm . . . . . . . . . . . . 52.1.2 Synchronization . . . . . . . . . . . . . . . . . . . 62.1.3 Hardware versus software timestamping . . . . . . 82.1.4 The terms accuracy and precision . . . . . . . . . 8

2.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . 92.2.1 Hirschmann . . . . . . . . . . . . . . . . . . . . . 92.2.2 Study at Tampere University of Technology . . . 92.2.3 Hardware Tests . . . . . . . . . . . . . . . . . . . 9

2.3 Testing Environment . . . . . . . . . . . . . . . . . . . . 102.3.1 Hardware . . . . . . . . . . . . . . . . . . . . . . 10

2.3.1.1 Intel Galileo . . . . . . . . . . . . . . . . 102.3.1.2 Storage device for the Intel Galileo . . . 102.3.1.3 Wiring . . . . . . . . . . . . . . . . . . . 10

2.3.2 Software . . . . . . . . . . . . . . . . . . . . . . . 112.3.2.1 Build system . . . . . . . . . . . . . . . 112.3.2.2 Build environment . . . . . . . . . . . . 112.3.2.3 The Linux PTP Project . . . . . . . . . 112.3.2.4 Stress-ng . . . . . . . . . . . . . . . . . 12

iii

iv CONTENTS

3 Method 133.1 Preparing the hardware . . . . . . . . . . . . . . . . . . . 13

3.1.1 Building the custom Linux image . . . . . . . . . 133.2 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4 Results 164.1 Reading the Charts . . . . . . . . . . . . . . . . . . . . . 16

4.1.1 Calibration . . . . . . . . . . . . . . . . . . . . . 164.2 No Load . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.3 50% CPU Usage . . . . . . . . . . . . . . . . . . . . . . 184.4 100% CPU Usage . . . . . . . . . . . . . . . . . . . . . . 194.5 I/O Stress . . . . . . . . . . . . . . . . . . . . . . . . . . 204.6 Network Stress . . . . . . . . . . . . . . . . . . . . . . . 214.7 OS Stress . . . . . . . . . . . . . . . . . . . . . . . . . . 224.8 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . 23

5 Discussion 255.1 CPU heavy tests . . . . . . . . . . . . . . . . . . . . . . 255.2 I/O, OS and Network tests . . . . . . . . . . . . . . . . . 265.3 Reliability and testing environment . . . . . . . . . . . . 26

5.3.1 Synthetic tests . . . . . . . . . . . . . . . . . . . 275.4 Improvements and future research . . . . . . . . . . . . . 27

6 Conclusions 28

References 29

Appendices 31

A Preparing SD Card for Linux 32

B Building a Linux Distribution for the Intel Galileo 33

C Configurations of PTP Implementations 35

D Data Extraction 36

Chapter 1

Introduction

Many modern computer systems are completely reliant on processes be-ing finished on specified real-time deadlines. Whenever missing a real-time deadline causes a total system failure, we call this a hard real-timesystem[1][2][3]. These types of systems serve a wide variety of functionsin many different fields ranging from industrial applications, to comput-erized financial systems[2][1]. The need for hard real-time systems raisesthe need for accurate time synchronization methods between microcon-trollers in distributed systems and there are several different widely usedtime synchronization protocols available with a range of different accu-racies possible[1].

The IEEE 1588 standard defines one of these protocols called PTP,Precision Time Protocol. PTP is used for very accurate time synchroniza-tion in the sub microsecond range usually over a local area network[4].The IEEE 1588 standard specifies a minimal required accuracy for imple-mentations of the protocol as “microsecond to sub-microsecond accuracyand precision”[4]. IEEE 1588 performance has been measured both inreal world tests using hardware as well as in software simulations[5][6][7].

This project aims to determine how reliable time synchronization be-tween microcontrollers is using software implemented PTP. This is doneusing two Intel Galileo development boards running Linux systems, con-nected with a category 5 ethernet cable, and software simulating systemload.

1

2 CHAPTER 1. INTRODUCTION

1.1 PurposeDue to the constraints placed upon hard real-time systems, time syn-chronization must have a strict margin of error or it could cause a totalsystem failure. This study will examine the effects different types ofsystem load could have on the accuracy of software implemented PTP.

1.2 Problem StatementThis report measures how much PTP synchronization accuracy and pre-cision is affected by system load using two Intel Galileo developmentboards. The goal is to determine if the accuracy and precision of soft-ware implemented PTP deteriorate under certain types of system load.

1.2.1 Research questionDoes software implemented PTP synchronization accuracy and precisionsuffer from system load?

1.3 ScopeThere exists other protocols such as NTP and GPS to synchronize timebetween devices, but they will not be tested in this report. The projectbudget limits the number of PTP-devices in our test, and as such, thetests will only involve two Intel Galileo development boards.

None of the tests in this report will be using hardware acceleratedPTP and as such, hardware acceleration of PTP will not be described indepth in the background chapter.

The terms IEEE 1588, PTP, and Precision Time Protocol will allbe used interchangeably to refer to the IEEE 1588-2008 standard. Thisreport will not be concerned with the deprecated IEEE 1588-2002 stan-dard.

1.4 OutlineThis report is divided into six chapters. This first chapter, Introduction,gives an overview of the project including purpose, problem statement,and scope. The second chapter, Background, gives a brief background of

CHAPTER 1. INTRODUCTION 3

the technologies used in creating this report; the PTP protocol as wellas any hardware and software used in testing. Select related works arealso presented in this chapter. The third chapter, Method, describes howthe testing environment was set up and how testing was carried out. Inthe fourth chapter, Results, the results of the tests are presented. Theresults are then discussed in the following and fifth chapter, Discussion.The last and sixth chapter, Conclusions, presents any conclusions drawn.

References can be found following the last chapter, followed by ap-pendices describing technological details of the Background and Methodchapters.

Chapter 2

Background

This chapter gives an introduction to all of the technologies involved inthis report. A background to the Precision Time Protocol (PTP), includ-ing how it works in practice, is given. After PTP have been introduced,work related to our research question is presented. Finally, the testingenvironment, including both hardware and software used, is presented.

2.1 The Precision Time ProtocolThe aim of the Precision Time protocol (PTP) is to provide a administra-tion free, fault tolerant, sub microsecond accuracy, standardized protocolfor synchronizing clocks in distributed systems within a local area net-work. PTP was designed for environments where using GPS would betoo expensive or simply not possible due to connectivity issues. In 2004,it was tested in practice, and achieved an accuracy of roughly 50 ns be-tween the connected devices. PTP is used mostly in industrial tests andautomation environments[1]. Compared to other time synchronizationprotocols, it is still a relatively new and a couple of different implemen-tations currently exist in use. PTP was originally defined in the IEEE1588-2002 standard in 2002. In 2008 a revised standard, IEEE 1588-2008,was released with minor improvements.

IEEE 1588-2008 defines PTP to use a master-slave hierarchy for clocksynchronization in a network. The network elects a grandmaster clockusing the best master clock (BMC) algorithm, and all the other devicesbecomes slaves to that master[4]. There may only be a single grandmasterin a network, but multiple master clocks are allowed. PTP defines threedifferent clock types: ordinary clock, boundary clock and transparent

4

CHAPTER 2. BACKGROUND 5

clock. An ordinary clock has a single PTP port while a boundary clockhas multiple PTP ports. The sole purpose of a boundary clock is toconvey clock synchronization to multiple ordinary clocks, from a singlemaster clock. The boundary clock also refreshes the timestamp on eachpacket sent from the clocks master, in comparison to a transparent clockwhich does not[4].

Figure 2.1: Illustration of a PTP network

2.1.1 Best master clock algorithmThe best master clock algorithm (BMCA) always runs on all boundaryand ordinary clocks in the network. Being run constantly, the clockis able to adapt to dynamic changes in the network, e.g. in case of agrandmaster change or partial blackout in the network. BMCA usesdata sets consisting of fields from two clocks to compare which clock is

6 CHAPTER 2. BACKGROUND

more eligible for being a grandmaster clock. Whenever a new potentialgrandmaster enters the network, all clocks will compare the data set fromthe old grandmaster with the new candidate. BMCA compares every fieldone by one, displayed below in decreasing precedence, only comparing thenext field if the previous fields in the data set are identical[4].

Priority One A field specified by the user ranging from 0 to255, where a higher priority takes precedence inthe BMCA.

Clock class The type of clock used by the device, rangingfrom GPS synchronized clock or a atomic cali-brated clock, to a slave only clock. A clock ofthe highest class shall never be used as anythingother than a grandmaster clock[8].

Clock accuracy The accuracy of the clock specified in units oftime, ranging from an accuracy of 25ns to an ac-curacy of less than 10 seconds[4].

Clock variance Static statistics of the variance and stability ofthe clock.

Priority Two Just like the priority one field, but with lowerprecedence.

Unique Port ID Often the physical MAC address of the port, thisis only used in case of a tie and has no actualrelation to the accuracy of the clock.

This way, two different devices comparing the same datasets (device Acomparing itself to device B and vice versa) will always come to the samedecision of which clock should be the grandmaster.

2.1.2 SynchronizationAfter a master clock has been chosen, the synchronization may begin.According to IEEE 1588-2008 the master clock may broadcast the currenttime as often as 10 times per seconds to all its slaves. A time broadcastusually consists of 3 messages in total, two from the master and one fromthe slave as shown in figure 2.2.

CHAPTER 2. BACKGROUND 7

Figure 2.2: Synchronization flow

The master starts with sending a Sync message at time T1, and allreceiving clocks stores the time T1’ when the message was received. Ifthe master has PTP support in the network hardware, the Sync messagewill also contain the time T1, otherwise a Follow_Up message will bebroadcasted to all slaves, which includes the time T1. Now all slaves havethe correct time of the master, and only the delay of the transit time ofthe packets in the network is left to calculate. The slaves respond to themaster with a Delay_Req message, timestamping the time sent as T2.The master responds to all requests with a Delay_Res packet, containingthe time the master received the delay request packet, T2’. All slavesare now able to calculate the transit time delay, the time it takes for amessage travel the distance from the master to a slave, and its own offsetrelative to its master. The time offset is defined as the difference betweena slave’s clock and its master’s clock. Let d be the transit time, and ωbe the offset between the two clocks. Since we assume the transit time

8 CHAPTER 2. BACKGROUND

is constant[4], we may calculate ω as follows.

ω + d = T1′ − T1−ω + d = T2′ − T2

ω = T1′ − T1 − T2′ + T22

The slave calculates its offset relative to master clock, and then adjusts itsown clock accordingly. Since the synchronization may happen a coupleof times per second, clock wander or jitter is assumed to be negligible[4].

2.1.3 Hardware versus software timestamping

Application Layer

Physical Communication Path

Linux KernelNetwork Stack

Physical Layer

HardwareTimestamping

SoftwareTimestamping

Figure 2.3: Methods of timestamping

There exists multiple ways of timestamping in PTP. In a scenario wheresoftware timestamping is used, the time spent in the Linux kernel networkstack is not accounted for, as shown in figure 2.3, but instead added to theoffset between the master and slave. This increases the offset by 2 ordersof magnitude[6]. In this report, only software implemented timestampingwill be considered.

2.1.4 The terms accuracy and precisionThe terms accuracy and precision are used extensively throughout thisreport. Wikipedia defines these two terms as “Accuracy is the proximity

CHAPTER 2. BACKGROUND 9

of measurement results to the true value; precision, the repeatability, orreproducibility of the measurement”[9]. In our report, the word accuracydescribes how close to zero the synchronization offset between a masterand slave is. Precision refers to how consistent the running average syn-chronization offset is. In the context of a PTP implementation, the wordperformance refers to both accuracy and precision.

2.2 Related WorkThis section presents work related to our research question “Does soft-ware implemented PTP synchronization accuracy suffer from systemload?” presented in chapter 1, Introduction.

2.2.1 HirschmannHirschmann, a German company specializing in industrial networkinghas released a white paper providing “an overview of the applicationpossibilities and function of the Precision Time Protocol”[6]. The paperpresents PTP accuracies of “about 5 to 50 µs as a pure software solution”.

2.2.2 Study at Tampere University of TechnologyA study from The Institute of Digital and Computer Systems at Tam-pere University of Technology in Finland analyzed the results of softwareimplemented PTP over a Wireless Local Area Network (WLAN) whichalso included results using a direct ethernet connection between two lap-tops. The tests presented in the study shows an average PTP accuracyof 7 µs between the two laptops with a variance of 10.8 µs2[7].

2.2.3 Hardware TestsProfessor HansWeibel at Zurich University of Applied Sciences in Switzer-land published an article in the Embedded World journal in 2005 ana-lyzing accuracies using hardware implemented PTP. The study showssub 100 nanosecond accuracies with “the significant variation [...] within±80 ns”[5].

10 CHAPTER 2. BACKGROUND

2.3 Testing EnvironmentThis section presents information regarding the testing environment usedduring the creation of this report. All software, hardware, and configu-ration needed is presented. Wherever needed, additional details found inappendices will be referred to.

2.3.1 HardwareThis section contains short descriptions about all hardware used in thisstudy.

2.3.1.1 Intel Galileo

The Intel Galileo (from here on referred to as Galileo) is a developmentboard designed by Intel Corporation. Developed for use in “Internet ofThings” applications, the Galileo is equipped with a Intel Quark X1000CPU[10]. The Quark X1000 is a 32-bit x86 system-on-a-chip CPU run-ning at 400 MHz. The Galileo is fitted with 256 MB RAM. Both ofthe Galileos used in testing ran on official Intel Galileo firmware version1.1.0.

2.3.1.2 Storage device for the Intel Galileo

The Intel Galileo uses a microSD card as its storage device. The microSDcard should be formatted with a single bootable primary partition span-ning the whole device, starting at sector 1024. The disk is to be formattedas a Master Boot Record (MBR) disk with a single FAT32 partition. Onmost Linux distributions including Ubuntu 16.04, a tool called fdisk maybe used for partitioning the SD card. For more information see appendixA.

2.3.1.3 Wiring

To connect the Galileo boards together, short distance category 5 ether-net cables are used. To monitor and connect to the devices, TTL 232RUART to USB cables are used.

CHAPTER 2. BACKGROUND 11

2.3.2 SoftwareFirst in this section is a short introduction of the build system and buildenvironment. Then, the software used in the testing is presented.

2.3.2.1 Build system

The build system used for this project is The Yocto Project[11]. TheYocto Project describes itself as “an open source collaboration projectthat provides templates, tools and methods to help you create customLinux-based systems for embedded products”[11].

The build engine in the Yocto project is called Bitbake. Bitbake isan asynchronous generic task executor used for handling dependenciesand cross compilation in the Yocto project[12]. Bitbake prepares andassembles a cross compiler toolchain to compile the custom Linux distri-bution used in the testing of PTP. The cross-compiler collection used inthis project was GCC1[13], since the Linux kernel is tied to GCC due tousage of GCC specific extensions to the C programming language.

2.3.2.2 Build environment

The build environment chosen was Ubuntu Linux 16.04[14], as the Yoctoproject officially support Ubuntu 16.04[11]. Ubuntu 16.04 is a stable longtime support distribution, and still receives hardware and maintenanceupdates[14]. No issues were encountered with compiling the custom dis-tribution with Ubuntu 16.04, and thus Ubuntu 16.04 was used as buildenvironment throughout the testing. Ubuntu 16.04 was run inside a vir-tual machine for flexibility and for better encapsulation from unrelatedlibraries and other binaries which might interfere with the build system.

Both GNOME Boxes[15] and VirtualBox[16] were tested as virtualmachines for the project. Both were found to be capable enough for usethroughout this entire project.

2.3.2.3 The Linux PTP Project

The Linux PTP Project (linuxptp) [17] is a free and open source im-plementation of the IEEE 1588 standard for Linux systems. One of themost used implementations for Linux systems, the Linux PTP Project isofficially supported by Intel Corporation. The design goals of linuxptp

1GNU Compiler Collection

12 CHAPTER 2. BACKGROUND

are “to provide a robust implementation of the standard and to use themost relevant and modern Application Programming Interfaces (API)offered by the Linux kernel”[17]. The Linux PTP Project will be usedthroughout the testing as the reference implementation of PTP.

2.3.2.4 Stress-ng

Stress-ng is a stress testing program used to stress a computer in differentways[18]. Stress-ng features stress testing different computer subsystemsseparately. These subsystem specific tests can test against, but not lim-ited to, I/O, OS, Network and CPU. The tests of linuxptp will involvestressing these subsystems one at a time, while measuring the accuracyand precision of linuxptp. Linuxptp was compiled from source using thesame GCC toolchain used by bitbake when compiling in the build envi-ronment. Stress-ng was as chosen since due to its ease of use and thepossibility to create a static executable.

Chapter 3

Method

In this chapter all the steps taken to set up a testing environment as wellas carrying out the tests are described in detail. First, instructions forbuilding and booting the Linux image as well as configuring the softwareused are presented followed by the steps taken to carry out the testsdescribed in detail.

In this chapter, whenever possible, references to other sources as wellas to appendices are given for more in-depth technological detail.

3.1 Preparing the hardwareThe first step in setting up the testing environment was updating theGalileo firmware using the official Intel Galileo Firmware Updater[19].The Firmware Updater software was run on a Windows 10 PC with theGalileos connected directly over USB. Both of the Galileos were updatedto version 1.1.0.

3.1.1 Building the custom Linux imageThe process of setting up the environment for Bitbake and the stepsnecessary to compile the modified Linux kernel is described in appendixB.2. The compiled kernel, root filesystem and related files were subse-quently copied onto a 8GB FAT32 formatted microSD card as describedin appendix A.

13

14 CHAPTER 3. METHOD

3.2 TestingThe testing started when the custom Linux image with Linux PTP in-stalled on, was built. One of the Galileo boards were the designated slave,and the other were the designated master. The master and slave desig-nations did not change throughout the testing. Unnecessary processesrunning on the Galileo systems were killed to prevent irrelevant strainon the Linux kernel. The master was set up to broadcast Sync-messagesat an interval of 0.125 seconds between each message, for a total of eighttimes a second instead of the default of one Sync per second. IEEE1588 states that up to ten broadcasts may happen per second, thus eightbroadcasts per second was chosen to be able to gather data at an higherresolution without pushing the boundaries of the protocol.

A test consisted of connecting the two Galileo boards together throughan ethernet cable, and simultaneously starting both stress-ng and linux-ptp on both devices at roughly the same time. The linuxptp configurationand command line arguments remained unmodified for the duration ofthe testing, and can be found in appendix C. linuxptp logs the offsetbetween the master and slave for every Sync-message sent. This out-put is stored and logged, from which the offset data is then extracted asdescribed in appendix D.

An initial testrun was performed, synchronizing the two Galileos to-gether. The first testrun was performed without stress-ng running. Thetest ran for 45 minutes, recording more than 20 000 Sync messages. Theresults from the initial testrun were consistent enough that the length ofall subsequent tests were limited to a maximum of 15 minutes.

After that, a total of six different tests were performed. Each of thesix different tests all had a specific purpose of stress testing a single com-ponent of the device. The aim of a stress test is to measure how linuxptpperforms under circumstances where a subsystems resources are fully ex-hausted. This is to provide a general overview of which subsystems ofthe device affects the accuracy and precision of linuxptp the most.

No load The first test was a test without stress-ng run-ning, to provide a baseline of what is the averageoffset between the master and the slave in thecontext of software based timestamping.

50% CPU load Stress-ng was used to simulate a constant CPUusage of 50% in the second test. Only a single

CHAPTER 3. METHOD 15

CPU-heavy process was run.

100% CPU load The third test was much like the second test, butwith full load on the CPU for the duration of thetest.

Stress I/O The fourth test involved stress testing the Galileoboards’ I/O functionality together with the I/Oscheduler of the Linux kernel. Since linuxptp ismostly I/O based, this test was to test if heavyI/O load would affect the precision or accuracyof linuxptp.

Stress Network After the I/O test, a test stressing only the net-work itself was performed. The meaning of thistest was to stress the Linux kernel network stackto determine how big of a role the Linux ker-nel network stack performed in the accuracy andprecision of linuxptp.

Stress the OS The last test consisted of only testing the Linuxkernel, through various system call and virtualmemory management stress tests.

The time offsets reported by linuxptp was recorded and logged, the resultsare presented in the next chapter.

Chapter 4

Results

In this chapter, results from the tests described in the previous chapter,Method, are presented. Results are presented using scatter plots andline charts with complementing explanatory text. The first section givesan introduction on how to interpret the graphs presented in the chapter.Following that, results from the tests are presented in the following order:no load, 50% CPU load, 100% CPU load, I/O stress, network stress.Following the charts for all the tests, charts for comparison between testsare provided. For analysis of the results and commentary, please refer tothe following chapter, Discussion.

4.1 Reading the ChartsAll charts in this chapter display the PTP offset in nanoseconds on they-axes and time passed in seconds on the x-axes. Y-axes are all base10 logarithmic scales whereas x-axes are linear. All dots in the scatterplot represent one measurement of offset at one moment in time. Offsetis defined as the time difference between a slave’s clock and its master’sclock. There are also line charts integrated into the scatter plots whichare created using Gnuplot’s smooth bezier option to create a more hu-manly readable average. In the case of the comparion charts, only theline chart will be provided.

4.1.1 CalibrationWorth noting is that all charts start out with a very high offset thatstabilizes rapidly after approximately 20 to 30 seconds. This is due to

16

CHAPTER 4. RESULTS 17

PTP calibration and is not representative of actual performance.

4.2 No Load

1

10

100

1000

10000

100000

1x106

0 100 200 300 400 500

Off

set

(ns)

Time (s)

No load

Figure 4.1: No Load Offset

Figure 4.1 presents the data captured during testing with no systemload. With no system load, we can see that performance is relativelystable after initial calibration is finished. It is clear that most values staywithin a 100 − 150, 000 nanosecond range.

18 CHAPTER 4. RESULTS

4.3 50% CPU Usage

1

10

100

1000

10000

100000

1x106

0 100 200 300 400 500

Off

set

(ns)

Time (s)

50% CPU

Figure 4.2: 50% CPU Usage Offset

Figure 4.2 presents the data captured during testing with 50% CPU us-age. With 50% CPU usage, we can see that offset is a bit lower than withno load with the exception of the unusually high offset during calibration.Most values stay within a 100 − 125, 000 nanosecond range.

CHAPTER 4. RESULTS 19

4.4 100% CPU Usage

1

10

100

1000

10000

100000

1x106

0 100 200 300 400 500

Off

set

(ns)

Time (s)

100% CPU

Figure 4.3: 100% CPU Usage Offset

Figure 4.3 presents the data captured during testing with 100% CPUusage. Offset values, while not as stable as during no load, still mostlyfit within the 100 − 150, 000 range seen in figure 4.1 with no load.

20 CHAPTER 4. RESULTS

4.5 I/O Stress

1

10

100

1000

10000

100000

1x106

1x107

0 100 200 300 400 500

Off

set

(ns)

Time (s)

I/O Stress

Figure 4.4: I/O Stress Offset

Figure 4.4 presents the data captured during testing with I/O stress. It isclear that I/O stress gives very unstable offsets with values over 100, 000being the norm with four very short dips in offset value. We can alsosee how between seconds 330 to 360, offset stays over the 106 line forapproximately 40 seconds. This happens once again at the 400 secondmark but this time only for approximately 20 seconds.

CHAPTER 4. RESULTS 21

4.6 Network Stress

1

100

10000

1x106

1x108

1x1010

1x1012

0 100 200 300 400 500

Off

set

(ns)

Time (s)

Network Stress

Figure 4.5: Network Stress Offset

Figure 4.5 presents the data captured during testing with network stress.Values are less stable than with no load. There are also exceptionallyhigh offsets during calibration. Most offset values are clearly within the100 − 150, 000 range, just like in figure 4.1 containing data for no load.However, we can see that the values fluctuate a lot more during networkstress.

22 CHAPTER 4. RESULTS

4.7 OS Stress

1

10

100

1000

10000

100000

1x106

1x107

1x108

0 100 200 300 400 500

Off

set

(ns)

Time (s)

OS Stress

Figure 4.6: OS Stress Offset

Figure 4.6 presents the data captured during testing with network stress.With the exception of a relatively stable period of increasing offset valuesfrom seconds 100 to 270, values are very unstable.

CHAPTER 4. RESULTS 23

4.8 Comparison

1

10

100

1000

10000

100000

1x106

0 100 200 300 400 500

Off

set

(ns)

Time (s)

no load100% cpu

50% cpu

Figure 4.7: No Load / CPU Usage Comparison

Figure 4.7 presents the data captured during testing with no load, andCPU usage at 50% and 100%. With the exception of the extremely highoffset values during calibration for the test with 50% CPU usage, thetests with CPU usage perform slightly better than the test with no load.For the individual results of each test presented in this chart, please referto figures 4.1, 4.2, and 4.3.

24 CHAPTER 4. RESULTS

1

10

100

1000

10000

100000

1x106

1x107

0 100 200 300 400 500

Off

set

(ns)

Time (s)

no loadstress io

stress networkstress os

Figure 4.8: No Load / IO / Network / OS Comparison

Figure 4.8 presents the data captured during testing with no load, I/Ostress, network stress, and OS stress. We can clearly see that the onlystable results in this chart comes from the test with no load. All otherlines have fluctuations ranging from ± 6, 000 nanoseconds to ± 500, 000nanoseconds. For the individual results of each test presented in thischart, please refer to figures 4.1, 4.4, 4.5, and 4.6.

Chapter 5

Discussion

In this chapter, analysis and discussion related to the results presented inthe previous chapter, Results is given. First the results from the differenttests are discussed and analyzed. The chapter ends with a discussionon the reliability of the results, and various improvements for futureresearch. No charts will be presented in this chapter. For charts andresults, please refer to the previous chapter.

5.1 CPU heavy testsThe CPU heavy tests showed that in general, linuxptp was able to per-form more consistently with higher precision under no processor load,than under processor stress tests. In comparisons to the other tests,stressing the CPU decreased the average synchronization offset betweenthe master and the slave. The results presented in figure 4.7 showed amore accurate but less precise average offset compared to the no loadtest. This was not expected, and one possible explanation is how theLinux kernel process scheduler works. By context switching directly tolinuxptp from stress-ng, the kernel removes the need to sleep and wakeup the processor. This removes some of the start up cost from a con-text switch which linuxptp is able to profit from. Another explanationis processor affinity or scaling of the CPU clock frequency. The Linuxkernel may scale the processors’ clock frequency down in order to savepower if the processor is idling a lot. This hypothesis assumes that inthe test of no load, the Linux kernel scaled the CPU frequency down tosave power, which in turn affected the accuracy of linuxptp. Howeverthis can not be the answer since the Intel Quark X1000 a single core

25

26 CHAPTER 5. DISCUSSION

CPU, and is always running with a fixed clock frequency of 400MHz. Afull explanation requires further testing of the CPU and linuxptp whichis outside the scope of this thesis.

5.2 I/O, OS and Network testsThe visible dips in the I/O chart corresponds to when stress-ng changesstress test. Figure 4.8 shows that linuxptp performs terribly when the I/Osubsystem is under heavy load. During heavy I/O load the peak offsetsis as high as 1 millisecond, approximately 125 times worse than worstaverage offset during no load. From around 60 seconds to 280 secondsthe I/O tests heavily stress the virtual memory of the Galileo board,as writing and reading to virtual memory is tested by stress-ng. Theoffsets during the I/O test aligns with the offsets in the OS test duringthis period. This is because during this period in the OS test, stress-ngalso stresses the virtual memory through various system calls involvingmemory management. The correlation shows that the virtual memorymay form a bottleneck when under heavy load. Stress-ng switches fromtesting the virtual memory in the OS test (after 300 seconds) to testingother various system calls, linuxptp is able to slowly recover and lowerthe offset over the duration for the last 200 seconds.

The results in figure 4.8 also show that when increasing the load onthe network, the precision of linuxptp suffers. This shows that the Linuxkernel network stack affects software timestamping precision to a highdegree, as expected.

5.3 Reliability and testing environmentThe testing environment is far from ideal. The Galileo boards are lowperformance, single CPU core development boards. As such, the Linuxkernel can not separate computationally intensive processes (stress-ng)from high priority processes (linuxptp) by placing them on different cores.Running these tests on a multi-core CPU may produce vastly differentresults. The PTP network set up in these tests are also minimal, consist-ing of only two devices. A PTP network in the real world scenario ofteninvolves multiple masters and slaves in a tiered configuration as shownin figure 2.1.

CHAPTER 5. DISCUSSION 27

5.3.1 Synthetic testsThe stress tests from stress-ng does not correspond to real life scenarios.The synthetic tests have been designed to pressure the system to work atpeak performance. Stress-ng produce far too predictable and consistentload on the different subsystems of the computer to be considered equalto real world load on the system. The tests however was not designed tobe as close to a real world scenarios as possible, but only to show howlinuxptp performs under heavy load on different subsystems.

5.4 Improvements and future researchSeveral improvements can be made in future testing. For instance, record-ing which stress test stress-ng is currently running may provide additionalexplanations of the charts and the synchronization performance of linux-ptp. Redoing the tests on a multi core CPU might also provide beneficialinformation in how big of a role the kernel process scheduler plays in theperformance of linuxptp. Results from testing in a multi-core setup hasalso the possibility of testing the difference between a single vs multipleresource intensive processes per core which may also provide additionalinformation on the performance of software accelerated ptp implementa-tions. Changing the size of the PTP network may also provide additionalinformation on how system load affects the master clock separately fromthe slaves, which was not tested in this thesis due to hardware constraints.One additional improvement can be to test these results against a PTPnetwork with hardware timestamping capabilities, to give more decisiveresults on how the Linux kernel network stack affects the linuxptp underheavy load.

Chapter 6

Conclusions

Software implemented PTP is shown to stay within a 1 microsecond to 10microsecond range when the system is under no load, which is consistentwith the studies presented under Related Work in chapter 2.

Using software implemented PTP during heavy CPU usage is shownto improve accuracy slightly compared to the results from the test with-out any stress on the CPU. The reason for this increase in accuracy couldbe an effect of the design of the Linux kernel, however further researchis needed to verify this.

When the device is under heavy I/O or OS stress, the performanceof PTP suffer greatly. This degradation is probably due to a bottleneckforming on the device or in the Linux kernel. Further research is neededto verify the reason behind the degradation.

During heavy network stress, software implemented PTP is shownto occasionally have increased accuracy at the cost of precision. Thisshows that the Linux kernel network stack affects software timestamping,however the cause for this is not analyzed in this report and requiresfurther research.

These results brings us to the conclusion that the performance of soft-ware implemented PTP does degrade under certain types of system load,in this specific testing environment. Additional research is needed to de-termine the reasons behind this degradation of accuracy and precision,however the research question stated has been answered.

Finally, the results might not be applicable to real-world usage ofPTP. The tests were carried out using a minimal hardware setup and thestress put on the system was only a simulation, and not equivalent toreal-world usage.

28

References

[1] J. C. Eidson, Measurement, Control, and Communication UsingIEEE 1588. Springer-Verlag London, 2006, isbn: 978-1-84628-251-5. doi: 10.1007/1-84628-251-9.

[2] J. A. Stankovic, “Real-time computing”, Apr. 1992.[3] G. Buttazzo, Hard Real-Time Computing Systems, 3rd ed. Springer

US, 2011.[4] “IEEE standard for a precision clock synchronization protocol for

networked measurement and control systems”, IEEE Std 1588-2008(Revision of IEEE Std 1588-2002), pp. 1–269, Jul. 2008. doi: 10.1109/IEEESTD.2008.4579760.

[5] H. Weibel, “High precision clock synchronization according to ieee1588 implementation and performance issues”, Embedded World,vol. 22-24, p. 96, Feb. 2005.

[6] A. Dreher and D. Mohl, “Precision clock synchronization the stan-dard ieee 1588”, Hirschmann Automation and Control GmbH, Neckarten-zlingen, Germany, Tech. Rep., Rev 1.2.

[7] J. Kannisto, T. Vanhatupa, M. Hännikäinen, and T. D. Hämäläi-nen, “Precision time protocol prototype on wireless lan”, in Inter-national Conference on Telecommunications, Springer, 2004, pp. 1236–1245.

[8] G. M. Garner, “Use of IEEE 1588 Best Master Clock Algorithm inIEEE 802.1AS”, Technical Presentation, 2008.

[9] Accuracy and precision, WikiMedia Foundation. [Online]. Avail-able: https://en.wikipedia.org/wiki/Accuracy_and_precision(visited on 05/05/2017).

29

30 REFERENCES

[10] Intel Quark SoC X1000 Specifications, Intel Corporation. [Online].Available: https://ark.intel.com/products/79084/Intel-Quark-SoC-X1000-16K-Cache-400-MHz (visited on 05/05/2017).

[11] Yocto Project. [Online]. Available: https://www.yoctoproject.org/ (visited on 05/05/2017).

[12] Bitbake User Manual, Yocto Project. [Online]. Available: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html (visited on 05/05/2017).

[13] GCC, the GNU Compiler Collection, Free Software Foundation.[Online]. Available: https://gcc.gnu.org/ (visited on 05/05/2017).

[14] The leading operating system for PCs, IoT devices, servers and thecloud | Ubuntu, Canonical Ltd. [Online]. Available: https://www.ubuntu.com/ (visited on 05/05/2017).

[15] Boxes, GNOME Foundation. [Online]. Available: https://wiki.gnome.org/Apps/Boxes (visited on 05/05/2017).

[16] Oracle VM VirtualBox, Oracle Corporation. [Online]. Available:https://www.virtualbox.org/ (visited on 05/05/2017).

[17] The Linux PTP Project. [Online]. Available: http://linuxptp.sourceforge.net/ (visited on 05/05/2017).

[18] Stress-ng. [Online]. Available: http://kernel.ubuntu.com/~cking/stress-ng/ (visited on 05/05/2017).

[19] Intel Galileo Firmware Updater and Drivers. [Online]. Available:https://downloadcenter.intel.com/download/26417/Intel-Galileo-Firmware-Updater-and-Drivers (visited on 05/05/2017).

[20] Intel Quark SoC X1000 Board Support Package (BSP), Intel Cor-poration. [Online]. Available: https://downloadcenter.intel.com/download/23197/Intel-Quark-SoC-X1000-Board-Support-Package-BSP- (visited on 05/05/2017).

Appendices

31

Appendix A

Preparing SD Card for Linux

To format a microSD card for booting Linux, the sfdisk may be used,which is installed by default on Ubuntu 16.04. Make sure to substitute<DEVICE> for the device name in your Linux operating system (such assdb). This script assumes an SD-card with 8GB of storage, however sizemay be adjusted accordingly.

$ sudo sfdisk /dev/<DEVICE> << EOFlabel: doslabel-id: 0x3af14fc4device: /dev/<DEVICE>unit: sectors

/dev/<DEVICE>1 : start= 2048, size= 15351808, type=83EOF

After the SD card has been prepared and the Linux distribution compiled,the necessary files must be copied over. The files in the boot-folder andall its contents, bzImage which is the kernel itself, grub.efi, image-full-quark.ext3 which is the root filesystem, and core-image-minimal-initramfs-quark.cpio.gz which is the initial ramdisk used by the kernelduring boot should all be copied over to the SD card.$ find . -type f./boot/grub/grub.conf./bzImage./grub.efi./image-full-quark.ext3./core-image-minimal-initramfs-quark.cpio.gz

32

Appendix B

Building a Linux Distribution forthe Intel Galileo

B.1 Intel Quark Board Support PackageThe Intel Quark Board Support Package (BSP) archive[20], provided byIntel, contains sources and utilities for building a base distribution forthe Intel Galileo. In this project version 1.1.0 of the BSP archive wasused, since it was the latest archive containing all the required scriptsand the correct configuration.

B.2 Building stepsLines starting with “$” are considered commands for your favourite shell.Lines starting with “#” are considered comments.

$ wget http://downloadmirror.intel.com/23823/eng/\bsp_sources_and_docs_for_intel_quark_v1.1.0.zip$ unzip bsp*.zip$ 7z e *.7z$ tar xvf meta-clanton_v1.1.0-dirty.tar.gz$ cd meta-clanton_v1.1.0-dirty$ ./setup.sh -e meta-clanton-bsp$ source iot-devkit-init-build-env build# This lets you configure the kernel however you want$ bitbake virtual/kernel -c menuconfig# This builds and compiles everything, may take hours

33

34 APPENDIX B. BUILDING A LINUX DISTRIBUTION FOR THE INTELGALILEO

$ bitbake image-full

The kernel and related files will all be present in ./tmp/deploy/images/quark/.

Appendix C

Configurations of PTP Implemen-tations

C.1 Linux PTP Projectlinuxptp was run with a niceness of -20, to indicate to the Linux kernelthe importance of the process. This also makes the Linux kernel prioritizeCPU time allocation to prefer linuxptp (ptp4l), during heavy load on thesystem.

C.1.1 ptp.conf[global]logSyncInterval -3summary_interval -3

C.1.2 Command Line Optionsnice --20 ptp4l -Smql 6 -f ptp.config \-i enp0s20f6 2>&1 > /media/mmcblk0p1/ptp.log

35

Appendix D

Data Extraction

D.1 Linux PTP ProjectAn AWK script is used to extract time, offset, and delay from logfile.The script also subtracts initial offset value from all offsets to make timestart from 0:

# Get offset from first line{if (NR == 1) {

match($1, /[0-9]+\.[0-9]+/, a);shift = a[0]

}}

# Extract time with regex to ’a’{match($1, /[0-9]+\.[0-9]+/, a)}

# Print time, offset, and delay unless line contains words#"UNCALIBRATED" or "selected" for when PTP selects a new master# Subtract shift from offset to start from time 0!/UNCALIBRATED|selected/ {print a[0] - shift "\t" $4 "\t" $10}

To convert all numbers to their absolute value, all data was piped throughthe following command:

sed ’s/-//g’

36

www.kth.se