Neural Network Structure for Traffic Forecasting

William Zhuk

December 2019

1 Introduction

Traffic prediction is a long-standing and impor-tant field both customer-facing industry (Maps& Guidance Software) as well as control appli-cations (Intelligent Transportation Systems, ab-breviated in literature as ITS). The task is to usecurrent-time and historical data to predict fu-ture traffic, which then is used to re-route usersor alter controls like lights and freeway entranceregulators.

Historically, moving-average (ARIMA vari-ants) and statistical model driven approacheshave been leaders in this area. Today, the largeamount of public data from many traffic systemshave brought data driven Machine Learning ap-proaches to the forefront: LSTMs, node embed-dings, and spectral graph decompositions play amajor role in many algorithms.

Even more recently, Graph ConvolutionalNetworks have become a large part of learn-ing tasks that have easily accessible informationgraphs. These new approaches use a combina-tion of LSTMs and graph convolutions to encodeboth temporal and spatial features to help withprediction.

In this project, we investigate the usageof LSTM cells within convolutions, and howmuch expressive power is gained by this ap-proach instead of using the graph convolutionsand LSTMs in sequence. Future work can ex-pand on larger time horizon predictions, the dif-ference in methods on larger datasets, and po-tential deeper enhancements to in-convolutionLSTM-cells.

2 Related Work

2.1 Graph Convolutions

Graph convolutions have been introduced tomany problem areas lately to allow for the appli-cation of many convolutional techniques learnedin image processing into graph-based problems.Today, there are many different convolutions be-ing used, and in this paper, I have made use ofthe ”vanilla” GCN, GAT [5], and GIN [3] convo-lutions that are baked into pytorch-geometric.

The difference between these is the functionper convolution that applies post-aggregation togenerate the outputs before convolutions hap-pen again. GCNs hold a single set of weights,GATs have both weights and an attention mech-anism to increase expressiveness, and GINs havesmall feed-forward neural networks as universalapproximators within the convolutions.

2.2 Traffic Graph Convolution

The TGC framework I am using was shownin Cui’s paper [2], which also showed its rel-ative performance to previous models on theirdatasets. This model was centered around anovel Free Flow Matrix, which calculated whichstations were within range of causality for trafficbased on traffic limits.

FFRij =

{1, S∆t− Distij ≥ 0

0, otherwise(1)

where S is the speed of traffic flow, and ∆t isthe amount of time to look at the network. Thismask allows for more selective convolutions thatare guaranteed to not lose any precision for in-fluences on the road that move below a certainspeed. They then combine this with the graph’smodified (to include self edges) adjacency ma-trix

A = A + I (2)

Ak = min((A)k, 1) (3)

1

Figure 1: Cui’s TGC-LSTM Model [2]

for a certain number k steps, and finally multi-ply the end result by weights according to dis-tance. The operation can be seen in the leftpanel of Figure 1, using these results to selectfor and weight edges in the graph convolutionoperation.

This paper has the graph convolution beforethe LSTM cell, with the convolution’s outputfor every time step as the input for LSTM cells.The exact methods for implementing this arereliant on the small size of the graph (the loopdataset used is 323 detectors total), and are im-plemented by hand with matrix operations.

2.3 T-GCN

A similar model is found in the T-GCN pa-per [4]. This model was tested on many pre-diction horizons, proving stable on longer hori-zons. This paper uses the adjacency matrixand stacked convolutions to create a topologi-cally expressive model. Instead of looking at theroad network and adding edges for every sen-sor that can influence another sensor within acertain time period (Cui’s method above), themethod here uses repeated convolutions on theoriginal adjacency graph, bringing more distantnode information after every convolution.

The paper also went into some depth aboutthe model’s power against Gaussian and Pois-son noise the training distribution, showing thata model made via a series of convolutions thatfeed into an LSTM is robust under most real-world circumstances.

3 Dataset

Figure 2: Loop Dataset

The dataset used for this paper is LOOP, aset of time-series speed data from road installedmetal loop detectors in Seattle. The detectorsare located almost entirely on four connectedfreeways (I-5, I-405, I-90, and SR-520). Thishas data for average speeds across 5 minute seg-ments for a year, measured at 323 sensor sta-tions.

The benefits of the Loop dataset is that allsensors are very close to equivalent, and willhave similar noise profiles to each other. Sincethe area covered is also relatively small, manyexternal factors like weather will apply to allnodes in the dataset roughly uniformly.

The downsides to the dataset is the lack ofextra features beyond just average speeds, andthe lack of positional data about the sensors.The dataset already has free-flow-matrices pro-vided that used the distances to calculate whichnodes are reachable within certain time frames,but a dataset with real distances would be muchmore powerful.

Despite the lack of distances, since thedata has loop detectors listed by their highway

2

and mile-marker, some information can be re-extracted. When doing so, it became apparentthat the FFR matrices provided are not correct.Locations with very similar mile counts on thesame highway in the same direction frequentlyhad no connections, and adding back those con-nections gave large differences in the number ofedges in the array. To counteract this, I cre-ated some FFR matrices manually using someapproximations for road intersections and testedthe different possible matrices as a hyperparam-eter.

4 Free Flow Matrix

One of the main defining factors of Cui’s TGCnetworks is the Free-Flow Matrix. This can in-fluence the ”neighborhoods”, or equivalently theconvolutional filter size used by the graph convo-lutional algorithm. Since traffic is a real-worldphenomenon, free-flow matrices allow us to im-pose the physical limits on traffic propagationinto the model.

Figure 3: 2 Day Speed Snippet

In Cui’s paper, the Free Flow Matrix is setthis to the speed limit (60mph). However, whenanalyzing the data speeds are usually higherthan this. The speeds over these datapoints varya lot, with some reaching past 100mph. Look-ing at the LOOP dataset used in their paper,we can see that even the averaged traffic speedsare best upper bounded by 80 mph to be morereasonable.

Figure 4: 60mph (default) vs 80mph (alt)

However, when using this new FFM formodel training, our performance does not move.This hints that flow-speed of traffic may not bethe correct formulation of the speed at which ac-tual traffic flows. After further investigation, thebottleneck for this algorithm is the hop countthat the TGC will look at. Changing the hopcount changes model performance regardless ofselected speed, and it seems the free-flow ma-trix plays an insignificant part in this operation.For the custom-made model by Cui, increasingK from gave large decreases in speed, and a largeenough number caused the model to break dueto some form of gradient exponentiation in back-prop. Only once K was large enough could themultiplication by the FFR matrix have a con-siderable difference.

Figure 5: Changing the FFM (s & bl) has muchless impact than changing k. The two bandsformed are k = 2, and k = 3,4

Since traffic moves slowly through the net-work, the model still performs well regardlessof the hop-count or FFR additions due to theunderlying adjacency matrix in it.

3

LSTM

323 x 10

323 x 64

Graph Conv

ReLU

8 convs

Dense

323 x 32

323 x 1

ReLU

323 x 10

Graph Conv

ReLU

8 convs

323 x 32

LSTM

Dense

323 x 1

ReLU

323 x 10

Graph Conv on A^k

LSTM Cell

10 convs (seq len)

Dense

323 x 32

323 x 1

ReLU

Figure 6: Examples of Architecture Variants for LSTM placement Many of the widths of the outputs ofthe LSTM and Convs are hyperparameters, as are the number of convolutions and type of convolution.

5 Train/Valid/Test Split

Cui’s original paper with the TGC-LSTM modelused an interesting splitting technique, whereevery ordered sequence of input generated datawas created, shuffled, and then given to each set.This surprised me, as it meant that valid andtest data had many sequence entries that over-lap with train data, due to the random shuffling.

Train Test

60

58 58

54 54

57 57

53 53

51 51

40 40

38 38

44 44

50 50

54

Figure 7: One example each from a hypotheti-cal train and test set, with 90% of the sequencedata overlapping.

After changing the split to be truly sepa-rate, and re-running existing models alongsidemy own, the difference was minor (a very smallincrease in valid/test data error). It seems thatLSTMs did not overfit to the data, probably be-cause of their sensitivity to the first input, whichwas always different in the three sets regardlessof method.

This being said, all of my data comes from atrain distribution on the first 70% of the times-tamps, validation on next 20%, and test on thelast 10%. This makes sure that none of themodel is trained on data pulled form the vali-dation or test set.

6 Comparison Models

The results seen from Cui’s paper are custom-made and tailored to a small 323 node graph.However, to use this technique on larger graphs,pytorch’s geometric framework [1] can saveheavily on memory and computation cost. Itwill also serve as a way to validate the results ina meaningful comparison.

There are also some existing GNN convolu-tion types within pytorch geometric that can beapplied to this problem to validate that the needfor an LSTM cell during repeated convolutionsis necessary. To do this, I tested different con-volutions that featured LSTMs either before orafter in the network architecture.

The unfortunate downside to using pytorch-geometric was that for this small graph, the lackof graph batching slowed down the computa-tion compared to the existing matrix-based cus-tom solutions. Extensions to these methods thatwould flatten out both by graph batch and bynodes in the graph, preserving order, and un-flatten afterwards would help with speedup.

4

7 Hyperparameter Search

Once I had my own model for graph convolu-tions, I started hyperparameter search on thenew structures to see how closely I could ap-proximate the existing methods. This was doneon both GAT models and the new custom con-volution model. The following is an example ofthe results for a search on the custom model.

Figure 8: Custom Convolution Hyperparame-ter Search Results. Given the number of hyper-params, the associated features are not shownfor space concerns and instead we just see re-sults.

After the search, it became clear that thebest models in each of the categories were per-forming roughly equivalently on the data. Thismeans that a series approach: using LSTMsto capture the sequential information, and thenspreading that around through graph convolu-tions, had roughly the same power and holdingthe LSTM inside of the convolutional operator.

0.0006

0.0008

0.001

0.002

0.004

GAN Custom

GAN vs Custom Losses

Figure 9: Losses of best of GAN and best Cus-tom.

However, it is important to point out thatthe sudden spikes in some epochs during the se-ries approach suggests it may be more brittle,especially since this is reflected in other hyper-parameter values that perform well. This maybe interesting to look into, as perhaps my im-plementation has some oversight.

8 Conclusion

Neural Networks built on top of both GraphConvolutions and Recurrent Networks are cur-rently the best systems for short-term trafficprediction. The results found on the LOOPdataset found little to no difference in the effec-tiveness of similar networks that re-order whenthe spatial and temporal data intermingle. Net-works with an LSTM after the graph convolu-tions performed equally well as networks thathad LSTMs before the graph convolutions, andagain equally with networks that had LSTMcells within the convolutions.

Equally surprising was that using convolu-tions on the edge matrix (A) versus the FFR-based matrices (Ak

⊗FFR) produced little dif-

ference when there were enough convolutions inthe single-neighbor networks. It provides oneextra point of evidence that repeated neighbor-based convolving is good at expressing topolog-ical features of graphs.

5

9 Code

The code is located at https://bitbucket.org/williamzhuk/wz-final-project-2019/src/master/The original library was made by Zhiyong Cui, at https://github.com/zhiyongc/Graph Convolutional LSTMThere are modifications throughout the Code V2 library, but to make it more manageable for bothmyself and the reader, you may ignore changes there and instead move into more recent Code V3library that I have built. I have made many changes throughout, but Cui’s code can still be found(somewhat edited) in the model training file, in the models file above the newer convolutional models,and in the dataset preparation file.

References

[1] pytorch geometric. https://pytorch-geometric.readthedocs.io.

[2] Zhiyong Cui, Kristian Henrickson, Ruimin Ke, and Yinhai Wang. High-order graph convolutionalrecurrent neural network: A deep learning framework for network-scale traffic learning and fore-casting. arXiv preprint arXiv:1802.07007, 2018.

[3] Jure Leskovec Stefanie Jegelka Keyulu Xu, Weihua Hu. How powerful are graph neural networks?2018.

[4] Chao Zhang Yu Liu Pu Wang Tao Lin Min Deng Haifeng Li Ling Zhao, Yujiao Song. T-gcn:A temporal graph convolutionalnetwork for traffic prediction. IEEE Transactions on IntelligentTransportation Systems-2019, 2018.

[5] Arantxa Casanova Adriana Romero Pietro Lio Yoshua Bengio Petar Velickovic, Guillem Cucurull.Graph attention networks. ICLR 2018, 2017.

6