Design and Analysis of Split- andAggregated-Transport Control Protocol (SA-TCP)

for Smart Metering Infrastructure

Tarek Khalifa∗, Atef Abdrabou†, Kshirasagar Naik∗, Maazen Alsabaan∗, Amiya Nayak‡, Nishith Goel§∗ Tkhalifa, Knaik, Malsabaa@uwaterloo.ca, Department of Electrical and Computer Engineering, University of Waterloo, Canada

†Atef.abdrabou@uaeu.ac.ae, Department of Electrical Engineering, United Arab Emirates University, UAE‡ Anayak@site.uottawa.ca, School of Information Technology and Engineering, University of Ottawa, Canada

§ Nishith@cistech.ca, Cistech Limited, 210 Colonnade Road, Ottawa, Ontario, Canada, K2E 7L5

Abstract—This paper investigates the TCP’s congestion controlmechanism effectiveness for smart meters. We show that theclassic congestion control causes a high loss rate for metereddata and disrupts competing traffic flows in a shared network.

The paper introduces the performance analysis of our pro-posed Split- and Aggregated-TCP (SA-TCP). SA-TCP complieswith industrial bodies’ directions (e.g., by Hydro One NetworksInc., Canada) in terms of the architecture, specifically thedeployment of hardware devices called Regional Collectors (RC).We argue that RCs can function as transport layer aggregationdevices rather than routing devices. This modification reducespacket loss rate and improves throughput for Smart MeteringInfrastructure (SMI) traffic.

Furthermore, we analyze the performance of the smart me-tering communication network running SA-TCP in terms ofthroughput, loss rate and latency. Our analysis is validated byextensive NS-2 simulation.

I. INTRODUCTION

Smart Metering Infrastructure (SMI) is a major applicationin the smart power grid. It is perceived as the system thatenables control and data collection and analysis, having bi-directional communication in place between a utility collectioncenter and smart meters and sensors. The system is charac-terized by the deployment of an extremely large number ofdata sources. The sources typically produce data at low rates(e.g., one segment per round trip time) either periodically orin response to triggered events. Data packets are collected atthe utility data center for processing and decision making.

From the communication perspective, a massive volume ofdata flows through a shared network of limited bandwidth,with the data requiring guaranteed end-to-end reliability. Thissituation calls for a transport control protocol (TCP). However,because the traffic transmission rate of individual meterscannot be lowered, TCP congestion mechanism is renderedineffective. The volume of traffic here does not come froma single source, rather from a large number of scatteredsources. The lack of proper transport congestion control causesinstability and degraded performance, such as high packet droprate and low throughput.

We propose a TCP-friendly protocol called Split- andAggregated-TCP (SA-TCP) to enhance TCP performance. Our

proposal makes use of devices installed as part of SMI calledregional collectors (RC). RCs are typically installed on polesat preselected locations within a local area network to routethe meter readings in a defined area through gateways ordirectly through a wide area network (WAN) to the utilityprovider [1]. In our approach, instead of limiting those RCsto layer 3 functionality (i.e., routing), we argue that if theyoperate at layer 4 (transport layer), they will achieve betterperformance. Thus, unlike the normal setup in which metersand sensors communicate over separate TCP sessions withthe utility collector device, SA-TCP brings in the idea ofaggregating those individual TCP connections at RCs (we referto them alternatively as SA-TCP aggregators).

The paper provides a mathematical model for the proposedscheme. The goal is to capture the application behavior of ametering device. The model combines a Markovian model forthe application and TCP behavior of a single meter separately,and then conducts the analysis of superposition and interactionof several devices sharing the network by means of standardqueuing system analysis.

This paper makes the following contributions:• It shows that enabling intermediate devices (e.g., RCs)

to operate at TCP level (rather than being limited tolayer 3) provides enhancement to TCP congestion controlperformance, such as reduced retransmission rate andimproved throughput.

• It provides a mathematical model for the proposed proto-col, SA-TCP, which helps capture the SMI behavior. Themodel is based on Markovian and queuing models.

Researchers have investigated various communication layersof SMI. Lower layer issues (e.g., the choice of communicationtechnology, link and IP layers) are summarized in [2] and [3].

To the best of our knowledge, our previous work [4] was thefirst to highlight the special requirements and shortcomingsof TCP in the smart metering network. Allalouf et al. [5]address the congestion caused by the large volume of meters’data by performing hop-by-hop traffic reduction, assumingthat information is required at certain intermediate devices butnot beyond them. They further assume that such intermediate

IEEE SmartGridComm 2012 Symposium - Smart Metering Infrastructure Networks and Data

978-1-4673-0911-0/12/$31.00 ©2012 IEEE 139

devices can process data at the application layer. They proposea cross-layer routing protocol that chooses the path that max-imizes data reduction. Kim and Thottan [6], however, do notconsider congestion, rather they focus on latency, so they pointout all the factors that contribute to a packet delay in TCP.Accordingly, they propose a modification of TCP, avoidingthe unnecessary delays to ensure fast delivery, especially fordelay-sensitive data. The focus of Kim et al. in [7], on theother hand, is on the security aspect of TCP in SMI. Theyargue that symmetric cryptography with pre-shared keys isappropriate because of the relatively poor resources. Theyconsider our proposed SA-TCP protocol as a scalable one.However, they advise that certain modifications are necessaryto make it secure.

The rest of the paper is organized as follows: Section IIintroduces the smart metering communication network withemphasis on SA-TCP architecture and traffic model. Sec-tion III explains the impact of the large-scale application ofsmart metering on TCP performance. Section IV provides theanalytical formulation for SA-TCP. This is followed by themodel validation in Section V. Finally, Section VI providesconcluding remarks and future work directions.

II. SYSTEM MODEL

Figure 2 depicts the smart metering infrastructure withemphasis on SA-TCP architecture. The figure shows the inter-connections of the different components involved at transportlevel. Meters and sensors are the main source of traffic. Atlower layers, they may form local wireless mesh networksand connect through gateways to the shared wide area network(WAN) (or possibly the Internet) and eventually to the collec-tion center. At transport level, the metering devices connectto intermediate devices, called Regional Collectors (RCs) [1],which are to function as SA-TCP aggregators. In the follow-ing, we summarize the characteristics and assumptions aboutthe smart metering traffic and communication infrastructurecharacteristics [8].

• Reliable delivery of every report sent by a meter orsensor is required. Every report carries a unique pieceof information related to a certain meter and a certaintime duration.

• Smart meters and sensors are stationary nodes.• The sources are scheduled to submit data reports continu-

ously at pre-configured schedule. Thus, TCP connectionsare set up as long-lived to avoid connection setup andtear down overhead.

• Data aggregation techniques such as MEAN, MEDIAN,MAX or MIN are not applicable. The utility provideris interested in each measurement, rather than somestatistical summary of the data.

• Meters are configured with a routing protocol that enablesthem to route data to a collection center. Smart Metersalready support the full stack of TCP/IP [9].

III. OVERVIEW OF SA-TCP

A. Motivation

The large number of data sources causes problems at thetransport level. TCP’s congestion mechanism is based on theidea of reducing a source’s transmission speed once packetsstart to get dropped at some congested router. In SMI, thehigh traffic does not come from a single source, rather itcomes from a large set of sources, each transmitting at a lowspeed (e.g., one packet per round trip time). As such, reducingthe speed in response to congestion in the network is notviable, which leads to the failure of the TCP’s congestionmechanism. In other words, the problem is equivalent toreplacing one large traffic source with, for example, onehundred thousand small sources producing the same amount oftraffic. If congestion occurs in the network, the one source mayreduce its congestion window to one, which is the minimum.Nonetheless, if each of 100k sources reduces its congestionwindow to one, a total of window size of 100k results. Moredetailed explanations about the shortcomings of TCP in SMIare available in our recent work [10].

B. SA-TCP Mechanism

The proposed solution to the above shortcomings of TCPin a smart metering system is as follows:• the main idea is splitting the TCP connections at a

device called the SA-TCP Aggregator and forming aunified TCP connection to the data collector. Therefore,instead of there being one direct TCP connection betweenevery meter and the collector, there will be two-hop TCPconnections, the first to the aggregator ad the second fromthe aggregator to the collector.

• The TCP protocol at end points, including meters, SA-TCP aggregators and the utility data collector, is used asis without modifications.

• Transport session are long-lived, meaning there is nooverhead of connection setup and termination.

Figure 1 and Fig. 2 depict the idea of one-hop TCP andtwo-hop TCP. In Fig. 2, the regional collectors may take onthe task of performing TCP connection aggregation rather thanjust forwarding packets at IP level.

Aggregation of TCP connections may also seem similar to[11]. This work [11], however, was introduced for GeneralPacket Radio Service (GPRS) to enforce fairness and improvelink utilization. The aggregation there is in the form ofunifying the TCP state information (e.g., round trip time andcongestion window) of a set of TCP connections that a singlemobile device initiates.

C. Simulation of SA-TCP

To illustrate the benefit of re-designing the transport pro-tocol architecture to include an aggregator at TCP level,NS-2 is used as follows. The two networks of Fig. 1 andFig. 2 are plugged into the simulator NS-2 with 10,000meters to compare the two architectures while observing twoperformance measures, namely, packet delivery latency and

140

Fig. 1. One-hop TCP Connections

Fig. 2. Two-hop TCP Connections

packet retransmission rate. The simulation compares betweentwo extreme scenarios. The first one (Fig. 1) has no TCPaggregators, which (from a congestion control perspective)is equivalent to having as many aggregators as there aremeters, each acting as its own aggregator. The second scenarioemploys only one SA-TCP aggregator.

In the case of one SA-TCP aggregator, Fig. 4 shows agreat improvement in the retransmission rate, which dropsfrom around 120% to below 1%. However, as Fig. 3 shows,the latency has worsened. The percentage of delayed reportsbeyond the deadline went up from around 1% to 30%. Moresimulation results are available in [10].

The experiments suggest that latency and retransmission rateare two conflicting performance objectives. They suggest alsothat a good solution lies in between, by choosing the rightnumber of SA-TCP aggregators.

Fig. 3. Percentage of Delayed Meters’ Reports

Fig. 4. Retransmission Rate of Meters’ Reports

Fig. 5. Overview of SA-TCP Model

IV. MODELLING SA-TCP

In the following, we develop a mathematical framework tocapture the behavior of SMI at TCP level and to be able toanalyze the system’s performance. For example, given a largenumber of sources, we will be able to determine the rightnumber of RCs to achieve the desired performance (i.e., packetloss rate and end-to-end delay). The formulation takes intoaccount the major factors that impact performance, namely,meter application and TCP behaviour and network parameters,such as link capacity, buffering capacity and mechanism, andpropagation delay.

Figure 5 provides the big picture of the model. Two stagesform the end-to-end architecture. The first stage constitutesmeters sharing a local network and thus divided into regionalblocks (RB), with each block connecting to a single SA-TCPaggregator. The group of the aggregators forms the secondstage as they connect through a shared WAN network to asingle collector.

We model each of the RBs separately and then combineall the RBs statistically, given that traffic is assumed tofollow Poisson distribution. Each of the RBs is modelledas a number of sources (i.e., meters or sensors) generatingtraffic according to the congestion window mechanism andapplication behaviour. This traffic is subject to queuing atan intermediate bottleneck node (Fig. 6). More precisely, we

141

Fig. 6. Interaction between Sources and Network

Fig. 7. Meter Markov Model

model the sources by means of Markov chain modelling andthe bottleneck as an M/M/1/B queue model. The Markovchain and the queue interact as shown in Fig. 6. That is, theaverage load calculated by the Markov chain of individualsources is summed up and analyzed at the queue. The queuemodel computes, as a result, the queuing delay and probabilityof packet loss, which are fed back to the Markov model asinput (The SA-TCP aggregator blocks will be modelled in asimilar manner in the future). This methodology is inspiredby [12]. The interaction between the Markov and queuingmodels continues for a number of iterations until a stable stateis reached, giving the probability of packet drop and averageround trip time.

A. Meter Model

A TCP congestion window determines the number of seg-ments that are sent in a single Round Trip Time (RTT)duration. Therefore, segment generation is modelled as aMarkov process, in which the states correspond to the sizeof the congestion window in segments, and the transitions aredetermined by the average RTT weighted by the probability ofsuccess or loss. Fig. 7 shows how a Markov chain is designedfor a meter and an SA-TCP aggregator, respectively.

In the meter Markov model, it is important to considerboth layers, the application and TCP. A meter is expectedto transmit data at a low rate intermittently. The applicationbehavior is modelled as a process alternating between active(Ton seconds) and inactive (Toff seconds) periods. We modelthe transition rate between the active states (0, 1 and 2) and theinactive state and vice versa as exponentially distributed withparameters α = 1

TOF Fand β = 1

TON, respectively. Although

there is no traffic model for meters in the literature, we followthis approximation because it is close to the anticipated actual

activity. Additionally, by adjusting the ON/OFF durations,we can adapt the application’s behavior easily. When theapplication is active, segments are transmitted in accordancewith the meter congestion window size. Regardless of theTCP variant used, the meter’s congestion window is low, notexceeding two, for two reasons: i) the amount of data is low(Small TON ); ii) the inactive duration is long (large TOFF ),which causes the congestion window to reset to its initial size.

Further to the above discussion, state transitions can beexplained as in the following bullets:• For W=1 segment (state 1), q1 is the probability of

transmitting the one segment successfully and is equalto (1− Pm), where Pm is the probability of loss on themeter-aggregator side (subject to network congestion).

• For W=2 segments (state 2), assuming independentlosses, the probability of sending the two segments suc-cessfully is q2 = (1− Pm)2

• Assuming an exponential distribution of RTT ( = 2Tp +Tq) with average Tr, the transition rate becomes δ = 1

Tr,

where Tp is the propagation delay and Tq is the queuingdelay.

• The congestion window grows from 1 to 2 at rate δ,weighted by the probability of successfully delivering theone segment.

• In case of segment loss, the window/state moves from 1or 2 to 0 at the rate δ, weighted by the probability of notsuccessfully delivering either segment.

• Timeout is approximated to be 5Tr, bases on the assump-tion that TCP estimates timeout as the average RTT plus 4times its standard deviation. Since RTT is assumed to beexponentially distributed, average and standard deviationare the same (i.e., Tr). This way, the transition rate fromstate 0 to 1 is calculated as τ = 1

5Tr

We get the following equations from the Markov chain:(δ + β)π1 − τπ0 − απI = 0 (1)δq1π1 − (β + δ(1− q2))π2 = 0 (2)δ(1− q1)π1 + δ(1− q2)π2 − τπ0 = 0 (3)βπ1 + βπ2 − απI = 0 (4)π1 + π2 + π0 + πI = 1 (5)

We solve the above linear equations to find the stationaryprobabilities; specifically we are interested in π1 and π2,which correspond to the transmission of 1 and 2 segmentsrespectively. The average traffic generated by a single source(λi) is calculated by the following equation:

λi = δπ1 + 2δπ2 (6)The total traffic generated by all the meters that have the

same ON/OFF and RTT characteristics is the following:λt =

∑ni=1 λi (7)

B. Network Model

The second major part of each RB is the shared bottleneck,which represents the network. Traffic entering a bottleneck iscalculated as the summation of individual λs (λt =

∑ni=1 λi).

The two measures, loss rate and average delay, are requiredto characterize the network condition, and will be the input to

142

the Markov chain models. Thus, we examine various queuingmodels, but present in this paper only M/M/1/B. This queuingmodel is computationally less complex and fits the problemin hand.

The queue is characterized by three parameters: queuesize (B); input traffic, which is assumed to be Poisson withparameter λt (for the blocks of meters); and service rate µm.For the first stage queue, µm is determined to be the SA-TCPaggregator’s offered load. On the other hand, the second stagequeue service rate µa is computed as the link capacity (C)over the segment size (MSS) in bytes (i.e., C

MSS ). The secondstage parameters can be computed by computer simulations.Equations (8) and (9) calculate the loss rate and the expectedqueuing delay.

P = ρB(1−ρ)1−ρ(B+1)

(8)

E[Tq] = ( 1µ ). 1−(B+1).ρB+B.ρ(B+1)

(1−ρ)(1−ρB)(9)

where ρ is the queue utilization factor ( λµ ).

V. SA-TCP FRAMEWORK VALIDATION AND DISCUSSION

Validation is performed through extensive comparisons ofthe model’s performance results with the network simulatorNS-2. We validate the model presented in Fig. 6 for themetering LAN blocks (Fig. 5). Simulation Parameters aredetailed below. The focus is on three important measures,namely, i) throughput (or source offered load): the rate atwhich segments are produced by a meter, ii) packet loss rate:the probability of packet drop due to buffer overflow, andiii) packet latency: time duration for a packet to arrive at thedestination.

A. Validation of Meter Markovian Model

It is possible to validate the source model and networkmodel independently [13]. The meter Markovian model pre-sented in Fig. 7 takes two input parameters, packet drop rateP and round trip time RTT, to compute throughput. Therefore,validating the meter model independently form the network isaccomplished in the following manner. First, NS-2 simulationis performed to calculate P, RTT and throughput. pluggingthe simulated P and RTT into the Markovian model, we getan estimated throughput. Fig. 8 shows a comparison of thethroughput (offered load) that is produced by the simulationversus the one estimated by the Markovian model.

The validation is done for a large range of meters (15,000to 25,000) sharing a link of 1 Mbps bandwidth, 50 msecpropagation delay, and a 40 packet DropTail buffer capacity.Each meter is characterized by an average Ton/Toff of 100msec/1 minute. As the figure shows, the model is proven toprovide a close match, especially as the number of metersincreases.

B. Validation of Complete blocks

A block (RB) is a collection of sources sharing a networkand a single collector. The sources keep adjusting their sendingrate according to the feedback received from the queue model

Fig. 8. Meter Offered Load

Fig. 9. Block Offered Load

until a steady state is reached, in which there is no furtherchange in the performance measures: offered load (λt), lossrate(P) and latency(RTT). Typically, steady state is reachedin 10 to 20 iterations. Fig. 9, 10 and 11 show the accuracyachieved in calculating λt, P and RTT as a result of the interac-tion between the sources’ Markov model and the queue model.In this experiment, meters are independent, and each followsexponentially distributed durations of active and inactive statesof mean Ton = 100msec and mean Toff = 1 minute. Whileactive, A source produces data packets of size 240 bytes (a 40-byte header). The shared link is characterized by bandwidth =1 Mbps, buffer capacity = 40 MSS and propagation delay =50 msec. In the simulation tests, sources use FTP and run for12,000 seconds to ensure correct statistics.

C. Impact of Varying Number of Aggregators

Another experiment is performed to determine how thenumber of aggregators influences the SA-TCP protocol’s per-formance. In this experiment, the mathematical framework isused to model 240,000 meters distributed into regions evenly.Each region has an SA-TCP aggregator. We assume that each

143

Fig. 10. Block Loss Rate

Fig. 11. Block End-to-end Latency

block of meters shares a link of BW = 1 Mbps and Tp =50 msec. The aggregators share an E1 link (BW = 2 Mbps,Tp = 50 msec.) on the WAN side. We vary the numberof aggregators from zero to 15. Zero means no SA-TCPaggregators are used, which corresponds to the one-hop TCPsetup case.

Fig. 12 shows the performance results pertaining to end-to-end (i.e., meter to Utility Data Collector) packet loss rate andlatency. Clearly, the loss rate is high when no aggregators areused. As we add aggregators, the loss rate increases slightlyin the beginning (at number of SA-TCP Aggregators oneand two) but then decreases, approaching zero; nevertheless,latency increases. The slight increase is due to the limitedspace of buffers. As the number of aggregators increases, morebuffering capacity becomes available.

VI. CONCLUSIONS AND FUTURE WORK

Many recent research publications have identified variousissues of the TCP/IP transport layer in smart metering commu-nication infrastructure. In line with those, our work introducesSA-TCP as a new strategy to enhance TCP performance byletting congestion control operate as effectively as intended.We have shown how our proposed strategy is set up and, more

Fig. 12. Effect of Number of SA-TCP Aggregators

importantly, outlined a mathematical framework that modelsthe behaviour of TCP and SA-TCP, taking into account impor-tant factors such as the application behaviour, network settingsand the number of meters and RCs (acting as transport-level aggregators). This framework allows analysis of thethroughput, end-to-end delay and loss rate. The analysis haslead to interesting conclusions about how to further enhanceSA-TCP by varying the number of SA-TCP aggregators. Ourfuture work will extend this analytical framework to includethis case of having multiple aggregators.

REFERENCES

[1] D. Relich, “Smart meters on a roll in canada,” in Hydro One NetworksInc., Sep 2008.

[2] A. Zaballos, A. Vallejo, M. Majoral, and J. Selga, “Survey and perfor-mance comparison of AMR over PLC standards,” IEEE Trans. on PowerDelivery, vol. 24, no. 2, pp. 604–613, 2009.

[3] T. Khalifa, K. Naik, and A. Nayak, “A survey of communicationprotocols for automatic meter reading,” IEEE Communications Surveysand Tutorials, no. 2, 2011.

[4] T. Khalifa, K. Naik, M. Alsabaan, A. Nayak, and N. Goel, “Transportprotocol for smart grid infrastructure,” in IEEE ICUFN, Jun 2010.

[5] M. Allalouf, G. Gershinsky, L. Lewin-Eytan, and J. Naor, “Data-quality-aware volume reduction in smart grid networks,” in Smart Grid Commu-nications (SmartGridComm), 2011 IEEE International Conference on,oct. 2011, pp. 120 –125.

[6] Y.-J. Kim and M. Thottan, “SGTP: Smart grid transport protocol forsecure reliable delivery of periodic real time data,” in Bell Labs Tech.J., vol. 16, Dec. 2011, pp. 83 –99.

[7] Y.-J. Kim, V. Kolesnikov, H. Kim, and M. Thottan, “SSTP: A scalableand secure transport protocol for smart grid data collection,” in SmartGrid Communications (SmartGridComm), 2011 IEEE InternationalConference on, oct. 2011, pp. 161–166.

[8] E. C. Ltd, “High-level smart meter data traffic analysis,” in ENA-CR008-001-1.4, May 2010.

[9] Companion Specification for Energy Metering, “DLMS/COSEM archi-tecture and protocols,” in Green Book, DLMS UA, 1997-2007.

[10] T. Khalifa, K. Naik, M. Alsabaan, and A. Nayak, “A transport controlprotocol suite for smart metering infrastructure,” in Malaysia PowerElectronics, Industrial Electronics & Industrial Applications, Apr 2011.

[11] R. Chakravorty, S. Katti, I. Pratt, and J. Crowcroft, “Flow aggregationfor enhanced tcp over wide area wireless.” in INFOCOM, 2003.

[12] C. Casetti and M. Meo, “An analytical framework for the performanceevaluation of tcp reno connections,” Computer Networks, vol. 37, no. 5,pp. 669 – 682, 2001.

[13] A. Wierman and T. Osogami, “A unified framework for modeling tcp-vegas, tcp-sack, and tcp-reno,” in MASCOTS, oct. 2003, pp. 269 – 278.

144