A Data-scheduling Mechanism for Multi-homedMobile Terminals with Disparate Link Latencies

Farhan H. Mirani, Nadia BoukhatemComputer Science and Network Department

Telecom ParisTech, Paris, FranceEmail: {mirani, boukhatem}@telecom-paristech.fr

Minh Anh TranLaboratory of Algorithms, Complexity and Logic

University of Paris-Est (Paris 12), FranceEmail: minh-anh.tran@univ-paris12.fr

Abstract—A mobile terminal equipped with multiple networkinterfaces can often be found under coverage of multiple accessnetworks. Multihoming enables mobile terminals to communicatesimultaneously through multiple interfaces. Connection-orientedtransport layer mechanisms ensure reliable, in-order deliveryof data, however parallel data transfers in multihoming faceproblems due to out-of-order reception of data which causesfast-retransmissions and limit the communication throughput.

In this paper, we propose a transport layer based mechanismwhich stripes one flow’s data across multiple disjoint pathsefficiently, reducing possible occurrence of out-of-order data. Ourmechanism estimates arrival times of data packets to be sentthrough available paths in an association, and decides whichpackets are to be sent through each path. We compared ourmechanism with a relatively closer approach called WestwoodSCTP. We illustrate with the help of simulation results that ourmechanism proves to improve multipath communication.

I. BACKGROUND

Laptops and other mobile devices with multiple networkinterfaces are not rare sights these days. Devices are commonlybuilt with multiple network interfaces, usually of differentaccess technologies, both wired and wireless. On the otherhand, rapid growth in wireless and cellular service providershas enabled overlapping coverage of radio resources. A mobiledevice can easily be found in places where multiple serviceproviders strive hard to compete and gain more and more cus-tomer base. Apart from better connectivity, exploiting multipleinterfaces simultaneously can provide mobile devices withseveral cutting edge benefits: always-on ubiquitous access, re-liability, load-sharing, load-balancing, bandwidth aggregation,multi-criteria interface selection, etc [1].

Multi-homing in this context refers to a situation in whichmobile terminal is connected to more than one access networkssimultaneously. In a multihomed environment, available net-work resources can be aggregated by distributing out-goingdata over multiple paths. Data striping is a resource aggre-gation technique that improves application-end performancefor a multi-homed host by using simultaneously a variety ofavailable interfaces [2].

Data-striping at the transport layer best suits multi-homedmobile hosts because it is the lowest layer keeping end-to-endsemantics among peers. Also, unlike application layer, it doesnot have additional application layer data resequencing costs.This is due to the fact that the data received has already gonethrough resequencing at the transport layer [3]. However, data-

striping at transport layer introduces problems of its own, forexample, data-reordering at the receive-buffer. Since reliabletransport layer mechanism sorts data by assigned sequencenumbers, it becomes crucial that the data be received in order,preventing otherwise formerly stated problems.

In wireless mobile networks, conventional multipath datatransfer methods cause communication performance degrada-tion, which occurs because conventional methods stripe dataover multiple paths without the regard of their path character-istics, i.e. bandwidth, delay, error rates, etc. Consequently, onepath receives data much after the others, thus compromisingthe data sequence at the receiver. To avoid stalling commu-nication in such a scenario, multipath communication needshuge receiving buffers in order to allow receiver sufficientsorting time, which in turn, increases the end-to-end delay.Therefore, an intelligent data striping method is required toavoid performance degradation caused by the distinct pathcharacteristics in a multipath communication.

Stream Control Transmission Protocol (SCTP) [4] is atransport layer protocol with multi-homing support. ThroughSCTP, a multi-homed host can assign different IP addressesto multiple connected interfaces and use them in a singletransport layer connection, called an “association”. With itsoriginal settings, SCTP only allows a single path, or ‘primarypath’ to communicate in an association, while other pathsare kept dormant for backup purposes, like link failure ordata retransmissions. Simultaneous use of multiple paths fordata transfers can significantly improve system performancein terms of aggregated bandwidth, resource utilization as wellas system robustness. Other works on transmission of datathrough multiple paths in an SCTP association are presentedin [5] and [6].

In this paper, we propose a transport layer data schedulingalgorithm, named Forward Prediction Scheduling (FPS), whichaims at reducing data re-ordering at the receiver. FPS is anintelligent algorithm. It takes account of the delays incurredto the data during the course of its transmission and schedulesthe specific data transmission accordingly on each path so thatthe data received preserves its order.

Remainder of this paper is organized as follows. In Sec-tion 2, we discuss related works that have been presentedpreviously to address the issue, and the shortfalls in them.Section 3 elaborates the idea of our proposal of a scheduling

978-1-4244-3574-6/10/$25.00 ©2010 IEEE

algorithm, effectively reducing the number of data re-orderingat the receiver through forward prediction. Section 4 and 5contain analysis by simulation and conclusion respectively.

II. STATE OF THE ART

Striping, as defined by the authors in [7], is a generalpurpose technique for network resource aggregation. Authorsin [7] present a comparative performance analysis of data-striping over different levels of protocol stack. Data-stripingat ‘link layer’, also called ‘Inverse multiplexing’ has beenpresented in [8] and [9], which discuss methods for splitting,recombining and sequencing datagrams across multiple logicaldata links. Since inverse multiplexing techniques assume stablelink characteristics, such as bandwidth, latency and loss rates,the approach they opt to assign data to these logical linksis straight round-robin and applicable only to point-to-pointcommunication systems.

Another area of related works is the use of multiple pathsat transport layer to improve throughput and/or robustness ofthe end-to-end connections. Authors in [10] propose a solutioncalled mTCP. mTCP is an end-to-end transport layer protocolwhich aggregates bandwidth on several paths concurrently.mTCP separates the decision of when to send data, whichdata segment to send and which path to use to send thedata. The sender in mTCP is allowed to send new data ona path selected through a congestion window-based scoring-mechanism associated to each path. Another solution calledpTCP is proposed in [3]. pTCP works as a wrapper around aslightly modified version of original TCP called TCP-virtual(TCP-v). Each interface opens a TCP-v connection, bundledand managed by pTCP. pTCP performs striping based on thecongestion window size of each TCP-v connection. Stripingmechanisms addressed in [3] and [10] are viable and striping isperformed keeping in view the ratio of throughput of differentpaths. However effective in-order data delivery at the receiveris not ensured.

Another approach, Arrival-Time matching Load-Balancing(ATLB) [11] offers an intelligent data distribution method formultipath-TCP communication. It proposes a scoring mech-anism for each connection based on queuing delay of thetransmission buffer and the network path delay. Although [11]assigns calculated score to each path, data sequence willstill be compromised at the receiver side. This is becauseATLB treats the number of segments to send on each path, itdoesn’t treat which segment is to be sent. Sequence numbersof segments sent on different paths will always be closelycoupled and thus re-ordering will occur when higher-scorepaths will deliver segments much slower than the lower-scorepaths, causing data re-ordering.

Similarly, [5] propose CMT. With SCTP working at its base,CMT exploits multiple paths between multi-homed source anddestination hosts to increase throughput for an application.Multiple IP addresses can be bound to a single associationin SCTP. Since originally SCTP only allows data to be sentthrough a single path in an association, CMT offers a round-robin distribution of data over multiple available paths in an

association. The data is controlled by the size of the availablecongestion window of each path. Unlike CMT, LS-SCTP [6]stripes packets with more intelligence. Striping in LS-SCTPis governed by the ratio cwnd/RTT of each path. In-case ofvastly different path delays, multipath communication suffersby out-of-order packets at the receiver, and is dealt by increas-ing the receive-buffer size. However, increasing receive-buffersize will have an impact on end-to-end delay, which will affectthe multipath communication.

The closest work related to our research is presented in [12].WestwoodSCTP or W-SCTP performs source schedulingat transport layer and tries to improve the chances of in-order delivery of data over multiple paths. At the time oftransmission of each data segment, W-SCTP proposes tocalculate a score for each path based on the time it will takefor the segment to reach the destination. The path with thelowest score is then chosen to transmit the particular segment,and this repeats for all data segments to be sent until thecongestion windows of all the available paths are exhausted.It proposes a multibuffer − multihoming architecture, inwhich each available path has its own separate send-buffer(unlike SCTP, which has only one common send buffer).On the other hand, there is only one receiver buffer forthe association, which receives data from all the paths andtransfer it to higher layers. In addition to that, Selective ACKsor SACKs return on the path from which the last packetwas received, but a SACK can also carry information forsegments received through other paths as well (as in delayedacknowledgements). Upon reception of a SACK, all the sendbuffers in an association are refreshed to update information.For one thing, W-SCTP doesn’t control which packet to sendfrom a sequenced buffer, and as it will be explained with proofin the section of simulation results, W-SCTP can suffer seriousperformance degradation due to data re-ordering if the pathsin the association have different latencies. Also, since send-buffer is actual depiction of the receiver’s advertised-window,the authors in [12] don’t mention how to divide a commonsend-buffer into multiple, smaller send-buffers for each pathand don’t discuss any mechanism for management of thesebuffers.

So far we have established the idea that in order to achievemaximum striping performance out of a transport layer mul-tipath communication, data re-orderings at the receiver buffermust be kept minimal. Also, data re-orderings become criticalwhen multiple paths have different latencies. Resultantly somesegments arrive much earlier than others through quicker pathsand often cause head-of-line blocking problem.

III. FORWARD PREDICTION SCHEDULING ALGORITHM

A. Theory

As we have established the requirements for an efficientscheduling mechanism in a multipath communication, wepresent a solution to counter the shortfalls caused by paralleldata transfers. We propose ForwardPredictionScheduling(FPS), a striping mechanism which predicts packets’ arrivaltimes at the receiver, and then decides which packets to send

on a particular path, avoiding any possible re-orderings at thereceiver buffer.

Let us assume a multipath communication in which severaldisjoint paths exist among end points. These paths may havedifferent characteristics such as, bandwidth, delay and errorrate. Amount of data that can be transferred through eachpath in a given time depends mainly upon path characteristics.Some paths may carry data much faster than others, whichmay in turn cause system performance degradation. This isbecause the faster paths will expect the next packets of datato arrive shortly after the last packet received, and eventually insome time will send back an acknowledgement with missing-data reports to the transmitter and causing a retransmission.Our mechanism therefore targets on reducing number of re-orderings at the receiver and consequently avoids performancedegradation due to retransmissions.

As shown in figure 1, an intelligent striping module isintroduced at transport layer, allowing data to be distributedover more than one paths. Striping module estimates the end-to-end latencies of connected paths, and then transmits chosendata over a path.

Higher layers

Transport layer

Send-buffer

Striping module

Congestion control

Lower layers

Higher layers

Transport layer

Recv-buffer

Congestion control

Lower layers

Emitt

er s

ide

Rece

iver

sid

e

Fig. 1. Functional block diagram

In our mechanism, we focus on manipulating outgoing datain a manner that it reaches the receiver with minimum occur-rence of re-orderings. Since received data at transport layer isarranged by sequence numbers, our mechanism estimates theamount of data that can be transferred through different pathsin a given time duration and then decides which sequencenumber to send, such that the arrival of any sequence numberat the receiver is scheduled in order beforehand. In a multipathcommunication, the mechanism first calculates how much datacan be sent exactly before the arrival of the data through theslowest path, and then accordingly assigns sequence numbers.

B. Preliminaries

Consider a multipath communication where a transmittermay use simultaneously I paths to transfer data to the receiver.Each path i has particular characteristics:

• RTTi: Round-trip time of a packet on pathi. It is thetime between a packet leaves the send buffer and itsacknowledgement reception back at the transmitter.

• FTi: The time a packet takes from leaving the send bufferon pathi to being received at the receiver. Roughly halfof the RTTi.

• si: Size of the congestion window on pathi.

Also to avoid complexity, the packet size is kept equivalentto that of a Maximum Segment Size (MSS). Hence, thequeuing delay for one packet on each path is fixed. We denotethis variable by εi for each path i where,

εi =PacketSize

Throughput.

Thus, if one sends a transmission of si packets at time t onthe path i, these packets will be received respectively at timet + FTi + εi, t + FTi + 2εi, . . ., t + FTi + siεi and then, thecorresponding acknowledgements will be received respectivelyat time t + RTTi + εi, t + RTTi + 2εi, . . ., t + RTTi + siεi.

Congestion window size si increases with the successfulreception of an acknowledgement. Hence, a new transmissionwill start at time t + RTTi + siεi, as soon as the lastacknowledgement of the previous transmission is received.

C. Scheduling algorithm

In this section, we describe how our algorithm decideswhich packets to send at each transmission with the help of anexample. The concept can be understood by analysing Figure2. It shows a multipath communication consisting of twoactive paths i and j, with disparate latencies RTTi and RTTj

respectively where RTTj > RTTi. According to figure 2,pathj is ready to send new packets at time instance t. However,since pathi delivers packets quicker than pathj , packets onpathj will have to be scheduled. The packets to be sent onpathj will be advanced by the number of packets delivered onpathi before the reception on pathj . To achieve the target, wespecify following points,

• Let t′ be the reception time on pathj

• For each transmission on pathi, we calculate the numberof packets successfully delivered before the time instancet′

• We obtain the total number of packets delivered bysumming all the transmissions that start before timeinstance t′

Fig. 2. Multipath transmission and reception time intervals

Now let us calculate the number of packets successfullydelivered on pathi before the time instance t′. Let nl bethe number of packets delivered in a particular transmissionstarting at a time instance tl, where tl < t′. The reception

instants of packets are respectively given by tl +FTi +εi, . . .,tl + FTi + siεi.

nl will then be the biggest non-negative integer satisfying

nl ≤ si and tl + FTi + nlεi < t′. (1)

Note that if tl + FTi + εi > t′, all packets arrive after t′

and therefore nl = 0.Finally, N =

∑

tl<t′nl. (2)

where N is the total number of packets estimated to besuccessfully delivered before time instance t′.

Based on these calculations for N , pseudo-code presentedin figure 3 will further emphasize the idea of our mechanism’salgorithm. For every packet to be transmitted, the system willfirst check congestion window availability for the particularpath. Only once congestion window has allowed, it will esti-mate possible number of packets that can be delivered throughpaths other than the path-in-hand, before the packet-in-hand isestimated delivered. A packet then with calculated sequencenumber is transmitted and marked as sent. If, a particularpath’s congestion window doesn’t allow it to transmit anypackets, the next path is chosen for the same iterations, untilall the data has been sent.

While ( ! PktSent ) { If ( congestion window space ) { N = EstimateNbPktsDlvrdB4PktInHand ( ) TempSN = SN SN = SN + N If ( ! MarkedAsSent ( SN + 1 ) ) { Send ( ++ SN ) PktSent = TRUE MarkAsSent ( SN ) If ( N != 0) SN = TempSN } Else SN ++ } Else { If ( Path-i == Path-i -> last path ) { Path-i = Path-i -> first path } Else { Path-i = Path-i -> next path } } }

Fig. 3. Pseudo-code for scheduling algorithm

At each acknowledgement reception from pathi, the actualvalue RTTi of the corresponding packet is measured and thealgorithm should smoothly update the RTTi and FTi of thepathi [13]

RTTi ← αRTTi + (1− α)rtti.

Where α is a constant between 0 and 1 which represents theadaptation rate of the estimation. And then, FTi is estimatedby

FTi =RTTi

2.

IV. PERFORMANCE EVALUATION

In this section, we evaluate the performance of our mech-anism by simulation over Network Simulator 2 (NS-2) [14].We compared our mechanism to W-SCTP client striping dataover multiple paths. The comparison of the two mechanisms

show that the number of re-ordering is reduced significantlyusing our mechanism, specially in the condition where com-municating paths have different latencies.

A. Simulation scenario

For the simulations, it is assumed that the hosts H1 and H2already have a multipath communication established, and thatH1 has a FTP data stream to send to H2 by exploiting bothpaths simultaneously. To better understand the improvementsbrought by our mechanism, hosts are also configured with W-SCTP, for the same duration with the identical data streams.

The configurations used for comparison are different set ofpath delay ratios for the two paths in the association. Both W-SCTP and FPS are shown exploited with these configurationsand results are compared and explained likewise. Segment sizefor our simulations has been fixed to 1500 bytes each and thetopology is simulated for a duration of 30 seconds, whichsufficiently proves our claims.

B. Simulation results

1) Congestion window growth: In figures 4 and 5, wepresent the results to compare FPS with W-SCTP. FPS stripesdata on the available paths intelligently, with the help of end-to-end delay estimates incurred to data through network, hencecongestion window growth of both paths is shown stable assoon as they achieve a maximum level allowed by transportlayer mechanism. On the other hand, W-SCTP is shown tohave congestion window growth far from the ideal. Althoughpackets are sent by calculating score, sequence numbers ofthese packets are still closely linked. Since propagation delayon path-2 (W-SCTP2) is greater than on path-1 (W-SCTP1),path-2 delivers packets much after path-1 does. Meanwhilereceiver sends acknowledgements back to the transmitterthrough same paths from which it received the data packets,reporting missing sequence numbers. On reception of a thirdsuch acknowledgement, path-2 goes into fast-retransmit modeand its congestion window is halved. Since this behaviourprevails from the beginning of the communication, the cwndsize of path-2 fluctuates around the initial cwnd size (1 MSS).

0

10

20

30

40

50

60

70

0.0

3.0

4.7

6.3

7.9

9.7

11.3

13.0

14.7

16.4

18.1

19.8

21.4

23.1

24.8

26.5

28.1

Cwnd

(pkt

s)

Time (sec)

W-SCTP Path 1

W-SCTP Path 2

Fig. 4. Cwnd growth pattern with W-SCTP. Path delay ratio 1:4

2) Throughput: Figures 6 and 7 present throughput com-parison of both paths among W-SCTP and FPS. In FPS,throughput of both paths is shown stable after a certain levelallowed by transport layer mechanism. On the other hand, W-SCTP throughput degrades due to fast-retransmissions caused

0

10

20

30

40

50

60

70

0.0

2.9

4.5

5.9

7.3

8.6

10.1

11.5

12.8

14.2

15.6

17.1

18.5

19.8

21.2

22.6

24.1

25.4

26.8

28.2

Cwnd

siz

e (p

kts)

Time (Sec)

FPS1

FPS2

Fig. 5. Cwnd growth pattern with FPS. Path delay ratio 1:4

by out-of-order data reception which keeps the congestionwindow of the slower path to minimum, resulting in reducedthroughput.

0

1

2

3

4

5

6

7

8

0.0

3.3

5.1

7.1

8.9

10.9

12.7

14.7

16.6

18.5

20.4

22.4

24.3

26.2

28.1

Thro

ughp

ut (M

bps)

Time (sec)

WSCTP Path 1

WSCTP Path 2

WSCTP Path1+Path2

Fig. 6. Throughput achieved in W-SCTP with path delay ratio 1:4

0123456789

10

0.0

2.9

4.5

5.9

7.1

8.6

10.0

11.4

12.8

14.1

15.6

17.0

18.3

19.7

21.1

22.5

23.9

25.2

26.7

28.0

Thro

ughp

ut (M

bps)

Time (Sec)

FPS Path1

FPS Path2

FPS Path1+Path2

Fig. 7. Throughput achieved in FPS with path delay ratio 1:4

3) Total number of packets delivered: Figure 8 showscomparison of total number of packets delivered successfullyfrom transmitter to receiver with four different settings of W-SCTP and FPS mechanisms. W-SCTP1-6 and W-SCTP1-4signify W-SCTP settings with path delay ratios of 1:6 and1:4 respectively. Same delay parameters have been applied forthe two FPS settings as well. Samples have been captured foreach two seconds throughout the simulation time. Althoughpath delay ratios are as steep as 1:4 and 1:6 among paths,FPS clearly outperforms W-SCTP, and the total number ofsuccessfully delivered packets is much higher in FPS than inW-SCTP.

V. CONCLUSION

Reducing number of re-orderings in a multipath communi-cation is crucial. Data re-orderings cause fast-retransmissions

0

100

200

300

400

500

600

700

800

900

2 4 6 8 10 12 14 16 18 20 22 24 26 28

Num

ber o

f pac

kets

Time (Sec)

WSCTP1-6

WSCTP1-4

FPS1-6

FPS1-4

Fig. 8. Number of packets delivered by different mechanisms’ settings

and limit the throughput of the communication. Most tech-niques presented so far address either changing emitter be-haviour towards acknowledgements or manipulating out-goingdata transmission. We propose FPS, a scheduling mechanismthat predicts data packets’ reception times in advance, andthen stripes packets onto multiple paths in a way that thepackets are received in-order. While our solution is basedon original SCTP architecture, our mechanism is adaptiveenough to be implemented on more widely-used TransmissionControl Protocol (TCP) as well. Our proposed mechanism isvalidated using NS-2, and the compared results show thatFPS succeeds in minimizing the re-ordering occurrences ina multipath communication.

We are currently investigating a cross-layer approach whichconsiders layer 2 parameters reflecting the actual link qualityto improve the path selection of multipath communication. Ourresearch is aimed at a complete multi-homing mobile solutionand FPS makes its contribution to it.

REFERENCES

[1] T. Ernst, N. Montavont, and et al., “Motivations and scenarios for usingmultiple interfaces and global addresses,” Internet draft, 2008.

[2] A. Habib, N. Christin, and J. Chuang, “Taking advantage of multihomingwith session layer striping,” IEEE INFOCOM, proceedings, 2006.

[3] H.-Y. Hsieh and R. Sivakumar, “ptcp: An end-to-end transport layerprotocol for striped connections,” proceedings of the 10th IEEE ICNP,2002.

[4] R. S. et al., “Rfc 2960: Stream control transmission protocol,” IETF,2000.

[5] J. R. Iyengar, P. D. Amer, and R. Stewart, “Concurrent multipath transferusing sctp multihoming over independent end-to-end paths,” Networking,IEEE/ACM Transactions, 2006.

[6] M. L. Ahmed Abd et Al, Tarek Saadawi, “A transport layer load sharingmechanism for mobile wireless hosts,” in proceedings of the SecondIEEE Anual Conference on Pervasive Computing and CommunicationsWorkshop, 2004.

[7] C. Brendan, S. Traw, and J. M. Smith, “Striping within the networksubsystem,” IEEE Network, vol. 9, Issue 4, pp. 22–32, Jul/Aug 1995.

[8] K. Sklower, B. Lloyd, G. McGregor, and D. Carr, “Rfc 1717: The pppmultilink protocol (mp),” IETF, 1994.

[9] A. C. Snoeren, “Adaptive inverse multiplexing for wide-area wirelessnetworks,” 12, September 1999.

[10] M. Zhang, J. Lai, A. Krishnamurthy, L. Peterson, and R. Wang, “Atransport layer approach for improving end-to-end performance androbustness using redundant paths,” USENIX, 2004.

[11] Y. Hasegawa, I. Yamaguchi, T. Hama, H. Shimonishi, and T. Murase,“Improved data distribution for multipath tcp communication,” in pro-ceedings of the IEEE GLOBECOM, 2005.

[12] C. Casetti and W. Gaiotto, “Westwood sctp: load balancing over multi-paths using bandwidth-aware source scheduling,” IEEE VTC, 2004.

[13] P. Karn and C. Patridge, “Improving round-trip time estimates in reli-able transport protocols,” ACM SIGCOMM Computer CommunicationReview, 1987.

[14] “The network simulator ns-2, http://www.isi.edu/nsnam/ns.”