UCR Time Series Classification Archive

Please reference as:

Yanping Chen, Eamonn Keogh, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen and Gustavo Batista (2015). The UCR Time Series Classification Archive. URL www.cs.ucr.edu/~eamonn/time_series_data/

Funded by NSF IIS-1161997 II and NSF IIS 1510741

Welcome!Dear Colleague If you are reading this, you are interested in using the UCR Time Series Classification Archive. This archive is a

superset of, and completely replaces [5]. Both [5], and this current Archive were born out of my frustration with papers reporting error rates on a single dataset, and claiming (or implicitly suggesting) that the results would generalize [6]. However, while I think the availability of previous versions of the UCR archive has mitigated this problem to a great extent, it may have opened up other problems.

1) Several researchers have published papers on showing “we win some, we lose some” on the UCR Archive. However, there are many trivial ways to get “win some, lose some” type results on these datasets (For example, just smoothing the data, or generalizing from 1NN to KNN etc.). Using the Archive can therefore apparently add credence to poor ideas (very sophisticated tests are required to show small but true improvement effects [3]). In addition Gustavo Batista has pointed out that “win some, lose some” is worthless unless you know in advance which ones you will win on! [4].

2) It could be argued that the goal of researchers should be to solve real world problems, and that improving accuracy on the UCR Archive is at best a poor proxy for such real world problems. Bing Hu has written a beautiful explanation as to why this is the case [2].

In spite of the above, the community generally finds the archive to be a very useful tool, and to date, more than 1,200 people have downloaded the UCR archive, and it has been referenced several hundred times.

We are therefore are delighted to share this resource with you. The password you need available in this document, read on to find it.

Best of luck with your research.

Eamonn Keogh

Data FormatEach of the datasets comes in two parts, a TRAIN partition and a TEST partition.For example, for the synthetic control dataset we have two files, synthetic_control_TEST.txt and synthetic_control_TRAIN.txt.The two files will be in the same format, but are generally of different sizes. The files are in the standard ASCII format that can be read directly by most tools/languages. For example, to read the two synthetic control dataset s into Matlab, we can type…

>> TRAIN = load('synthetic_control_TRAIN'); >> TEST  = load('synthetic_control_TEST' );

…at the command line.

There is one data instance per row. The first value in the row is the class label (an integer between 1 and the number of classes). The rest of the row are the data values, and individual time series.

This instance is in class 1

This instance is in class 2

function UCR_time_series_test %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% (C) Eamonn Keogh %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%TRAIN = load('synthetic_control_TRAIN'); % Only these two lines need to be changed to test a different dataset. %TEST  = load('synthetic_control_TEST' ); % Only these two lines need to be changed to test a different dataset. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

TRAIN_class_labels = TRAIN(:,1);     % Pull out the class labels.TRAIN(:,1) = [];                     % Remove class labels from training set.TEST_class_labels = TEST(:,1);       % Pull out the class labels.TEST(:,1) = [];                      % Remove class labels from testing set.correct = 0; % Initialize the number we got correctfor i = 1 : length(TEST_class_labels) % Loop over every instance in the test set      classify_this_object = TEST(i,:);   this_objects_actual_class = TEST_class_labels(i);   predicted_class = Classification_Algorithm(TRAIN,TRAIN_class_labels, classify_this_object);   if predicted_class == this_objects_actual_class       correct = correct + 1;   end;   disp([int2str(i), ' out of ', int2str(length(TEST_class_labels)), ' done']) % Report progressend;%%%%%%%%%%%%%%%%% Create Report %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%disp(['The dataset you tested has ', int2str(length(unique(TRAIN_class_labels))), ' classes'])disp(['The training set is of size ', int2str(size(TRAIN,1)),', and the test set is of size ',int2str(size(TEST,1)),'.'])disp(['The time series are of length ', int2str(size(TRAIN,2))])disp(['The error rate was ',num2str((length(TEST_class_labels)-correct )/length(TEST_class_labels))])%%%%%%%%%%%%%%%%% End Report %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Here is a sample classification algorithm, it is the simple (yet very competitive) one-nearest% neighbor using the Euclidean distance.% If you are advocating a new distance measure you just need to change the line marked "Euclidean distance"%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%function predicted_class = Classification_Algorithm(TRAIN,TRAIN_class_labels,unknown_object)best_so_far = inf; for i = 1 : length(TRAIN_class_labels)     compare_to_this_object = TRAIN(i,:);     distance = sqrt(sum((compare_to_this_object - unknown_object).^2)); % Euclidean distance        if distance < best_so_far          predicted_class = TRAIN_class_labels(i);     best_so_far = distance;    endend;

>> UCR_time_series_test1 out of 300 done2 out of 300 done…299 out of 300 done300 out of 300 doneThe dataset you tested has 6 classesThe training set is of size 300, and the test set is of size 300.The time series are of length 60The error rate was 0.12

Sanity Check

In order to make sure that you understand the data format, you should run this simple piece of matlab code (you can cut and paste it, it is standard Matlab)

Note that this is slow “teaching” code. To consider all the datasets in the archive, you will probably want to do something more sophisticated (indexing, lower bounding etc)

In this package we have produced a Excel file that gives basic information about the datasets (number of classes, size of train/test splits, length of time series etc)

In addition, we have computed the error rates for:

• Euclidean distance• DTW, unconstrained• DTW, after learning the best constraint in the test set*

*Note that our simple method for learning the constraint is not necessary the best (as explained in the next slide).

Worked ExampleWe can use the Archive to answer the following question. Is DTW better than Euclidean distance for all/most/some/any problems? As explained in [4], if DTW is only better on some datasets, this is not very useful unless we know ahead of time that it will be better. To test this we can build a Texas Sharpshooter plot (see [4] for details).In brief, after computing the baseline (here, the Euclidean distance) we then compute the expected improvement we would get using DTW (at this stage, learning any parameters and settings), then compute the actual improvement obtained (using these now hardcoded parameters and settings).

When we create the Texas Sharpshooter plot , each dataset fall into one of four possibilities.

We expected to do worse, but we did

better.

We expected an

improvement and we got it!

We expected to do worse, and we did.

We expected to do better, but actually did worse.

In our worked example, we will try to optimize the performance of DTW, and predict its improvement (which could be negative), in a very simple way.

Expected Improvement: We will search over different warping window constraints, from 0% to 100%, in 1% increments, looking for the warping window size that gives the highest 1NN training accuracy (if there are ties, we choose the smaller warping window size).

Actual Improvement: Using the warping window size we learned in the last phase, we test the holdout test data on the training set with 1NN.

Note that there are better ways to do this (learn with increments smaller than 1%, use KNN instead of 1NN, do cross validation within the test set etc). However, as the next slides show, the results are pretty unambiguous even for this simple effort.

Texas Sharpshooter Plot [4]

Expected Accuracy Gain

Act

ua

l Acc

ura

cy G

ain

The results are strongly supportive of the claim “DTW better than Euclidean distance for most problems”

We sometimes had difficultly in predicting when DTW would be better/worse, but many of the training sets are tiny, making such tests very difficult.

For example, 51 is BeetleFy, with just 20 train and 20 test instances. Here we expected to do a little better, but we did a little worse.

In contrast, for 76 (LargeKitchenAppliances) we had 375 train and 375 test instances, and where able to more accurately predict a large improvement.

0.8 1 1.2 1.4 1.6 1.8 2 2.20.8

1

1.2

1.4

1.6

1.8

2

2.2

Expected Accuracy Gain

Act

ua

l Acc

ura

cy G

ain

12

3

4

5

6

7

8

9

10

11

12

1314

15

161718

19

20 21 22

23

242526

27

2829

30 31

32

3334 35

3637

38

39

4041

42

43 4445

4647

4849 50

51

52

5354

555657

58

5960

6162

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

7980

81

82

8384

85

86

0.8 1 1.2 1.4 1.6 1.8 2 2.20.8

1

1.2

1.4

1.6

1.8

2

2.2

Expected Accuracy Gain

Act

ual A

ccur

acy

Gai

n

Plane

Car

Synthetic Control

Gun-Point

CBF

Face (all)

OSU Leaf

Swedish Leaf

50Words

Trace

Two Patterns

Wafer

Face (four)

Lightning-2

Lightning-7

ECG

Adiac

Yoga

Fish (re

adme)

Beef

Coffee

OliveOil

CinCECG t

orso

ChlorineConcentration

DiatomSizeReductio

n

ECGFiveDays

FacesU

CR

Haptics

InlineSkate

ItalyPowerDemand

MALLAT

MedicalImages

MoteStrain

SonyAIBORobot Surfa

ceII

SonyAIBORobot Surfa

ce

StarLightCurves

Symbols

TwoLeadECG

WordsSynonyms

Cricket X

Cricket Y

Cricket Z

uWaveGestureLibraryX

uWaveGestureLibraryY

uWaveGestureLibraryZ

Non-Invasiv

e Fetal ECG Thorax1

Non-Invasiv

e Fetal ECG Thorax2

InsectWingbeatSound

ECG5000

ArrowHead

BeetleFly

BirdChicken

Ham

Herring

PhalangesOutlinesC

orrect

ProximalPhalanxOutlineAgeGroup

ProximalPhalanxOutlineCorre

ct

ProximalPhalanxTW

ToeSegmentation1

ToeSegmentation2

DistalPhalanxOutlineAgeGroup

DistalPhalanxOutlineCorre

ct

DistalPhalanxTW

Earthquakes

MiddlePhalanxOutlin

eAgeGroup

MiddlePhalanxOutlin

eCorrect

MiddlePhalanxTW

ShapeletSim

Wine

WordSynonyms

Computers

ElectricDevice

s

FordA

FordB

HandOutlines

LargeKitchenAppliances

Meat

Phoneme

RefrigerationDevice

s

ScreenType

ShapesAll

SmallKitchenAppliances

Strawberry

UWaveGestureLibraryAll

Worms

WormsTwoClass

(after plotting in Matlab, the code is in Appendix A, you can zoom in to avoid the visual clutter seen to the right).

Suggested Best Practices/Hints1. If you modify the data in anyway (add noise, add warping etc), please give the modified data back to the archive

before you submit your paper (we will host it, and that way a diligent reviewer can test your claims while the paper is under review).

2. Where possible, we strongly advocate testing and publishing results on all datasets (to avoid cherry-picking), unless of course you are making an explicit claim for only a certain type of data (i.e. classifying short time series). In the event you don't have space in your paper, we suggest you create an extended tech report online and point to it. Please see [4] (esp. Fig 14) for some ideas on how to visualize the accuracy results on so many datasets.

3. If you have additional datasets, we ask that you donate them to the archive in our simple format.4. When you write your paper, please make reproducibility your goal. In particular, explicitly state all parameters. A

good guiding principle is to ask yourself Could a smart grad student get the exact same results as claimed in this paper with a days effort?. If the answer is no, we believe that something is wrong. Help the imaginary grad student by rewriting your paper.

5. Where possible, make your code available (as we have done), it will makes the reviewers task easier. 6. If you are advocating a new distance/similarity measure, we strongly recommend you test and report the 1-NN

accuracy (as we have done). Note that this does not preclude the addition of other of tests (we strongly encourage additional test), however the 1-NN test has the advantage of having no parameters and allowing comparisons between methods.

7. Note that the data is z-normalized. Paper [7] explains why this is very important.

Suggested Reading1. Xiaoyue Wang, Abdullah Mueen, Hui Ding, Goce Trajcevski, Peter Scheuermann, Eamonn J.

Keogh: Experimental comparison of representation methods and distance measures for time series data. Data Min. Knowl. Discov. 26(2): 275-309 (2013).

2. Bing Hu, Yanping Chen, Eamonn J. Keogh: Time Series Classification under More Realistic Assumptions. SDM 2013: 578-586.

3. Hills, J., Lines, J, Baranauskas, E., Mapp, J. and Bagnall, A. Classification of time series by shapelet transformation. Data Mining and Knowledge Discovery Journal. ISSN 1384-5810, 2013.

4. Gustavo E. A. P. A. Batista, Xiaoyue Wang, Eamonn J. Keogh: A Complexity-Invariant Distance Measure for Time Series. SDM 2011: 699-710

5. Keogh, E., Zhu, Q., Hu, B., Hao. Y., Xi, X., Wei, L. & Ratanamahatana, C. A. (2011). The UCR Time Series Classification/Clustering Homepage.

6. Eamonn J. Keogh, Shruti Kasetty: On the Need for Time Series Data Mining Benchmarks: A Survey and Empirical Demonstration. Data Min. Knowl. Discov. 7(4): 349-371 (2003)

7. Thanawin Rakthanmanon, Bilson J. L. Campana, Abdullah Mueen, Gustavo E. A. P. A. Batista, M. Brandon Westover, Qiang Zhu, Jesin Zakaria, Eamonn J. Keogh: Addressing Big Data Time Series: Mining Trillions of Time Series Subsequences Under Dynamic Time Warping. TKDD 7(3): 10 (2013)

Shiyuan Liu, Li Lv.Thiago Santos Quirino, Mei-Ling Shyu Pierre-Franqois Marteau,Cosmin Bocaniala, Lancaster University.Yueguo Chen,Anthony K.H. Tung, Beng Chin Ooi, National University of Singapore.Vernon Rego, Vernon Rego. Purdue University.Mislav Malenica, Tomislav SmucMan Hon WONG, ZHOU Mi, The Chinese University of Hong Kong.Guillaume Bouchard Xerox Research Centre Europe.Hoa Vo and David Joslin Seattle University.Carlotta Orsenigo , University degli Studi di Milano.Dr. Paolo CiacciaXiaoqing Weng, Jiaotong University.Longin Jan Latecki and Qiang Wang, Temple University.Tony BagnallMichail Vlachos, IBM.Bernard HugueneySourav MukherjeeMarcos M. Campos (Oracle)Ludmila I. KunchevaEdward Omiecinski and Jun LiVictor Eruhimov, IntelRob JasperAndre CoelhoGernot HerbstVit NiennattrakulNozomi MatsudaFlavio Miguel Varejao and Idilio DragoHAORIANTO COKROWIJOYO TJIOEMolnar MiklosSteinn Gudmundsson and Thomas RunarssonNiall Adams and Sai Wing ManIsabel Maria Marques da Silva, Maria Eduarda da Rocha Pinto Augusto da Silva and Joaquim Fernando Pinto da CostaPanagiotis Papapetrou and George KolliosHuang TanSergio GuadarramaAlicia Troncoso LoraPyry AvistPeng-Yi LaiYong FuSoheil BahrampourLong Yao and Meng BoRobert Moskovitch and Yuval ShaharAbdellali KelilPuspadevi KuppusamyQun Dai and Songcan ChenLisa GralewskiMaria Teresinha Arns Steiner and Rosangela VillwockAmir Ahmad and Galvin BrownAbhijit Jayant KulkarniXingquan (Hill) ZhuAmol Deshpande and Qiang QiuVercellis Carlo and Gianni AlbertiPamela Nerina LlopTobias SchefferJochen FischerMao Ye and Yingying ZhuCintia LeraGeorge Runger and Rohit DasOmar U. Florez and Seungjin LimRuy Luiz Milidiu and Pedro TeixeiraMykola Galushka and Dave PattersonRahul SinhaMinh Hoai Nguyen and Fernando de la TorreEric EatonNGUYEN Van HanhLucas F. RosadaNicky Van ThuyneSkopal Tomas and Michal VajbarCarlo Piccardi and Martina MaggioMuhammad Aamir KhanLarry DeschaineJanosa AndrasAndrew StarkeyKarthik Marudhachalam and Ansaf Salleb-AouissiRayner Alfred and Samry Mohd Shamrie SaininEvins lio and Oumsis OumsisHanjing, SuMarco CuturiWang ZhiminMiao Zeng and Yubao LiuRene Vidal and Rizwan ChaudhrySusan Cheng and Min DingAmit Ganatra and Dhaval Bhoi

awei Han & Manish GuptaIinstry Liang and Qin LvEirik Benum RekstenLucas Gallindo Martins SoaresSungyoung Lee and La The VinhHuidong (Warren) JinSantiago VelascoJairo CugliariQuazi Abidur RahmanHungyu Henry Lin and James DavisAzuraliza Abu Baka and Almahdi Mohammed AlmahdiBenjamin Bustos and Victor SepulvedaKrisztian BuzaYoung XinShen Wang and Haimonti DuttaVictor Garcia-PortillaSamir Al-StouhiGuo FeiTeodor CostachioiuStephen PollardEser Kandoganlida rashidiTakashi Washio and Satoshi HaraMeizhu Liu and Baba C. VemuriJose Principe and Sohan SethRamanuja Simha and Rahul TripathRodolphe JENATTON and Francis BachSiladitya Dey and Ambuj SinghMahsa Orang and Nematollaah SHIRI V.Saket SaurabhWu WushDeepti DohareSubhajit Dutta.Evgeny PyatkovIon George TODORANQin ZouK.S.SUBHASHINIFeng Gu, Julie GreensmithPedro FelzenszwalbOnur SerefAlessia AlbaneseSeniha Esen Yuksel and Paul GaderAlexandros IosifidisRana Hussein and Neamat Farouk El-GayarMatteo DanielettoYing ZhangSaeid RashidiXiaobo Wu and S ChenSun Lei and Yu-Jiu YangBelen Martin BarraganMustafa Ahmed and Hua-Liang WeiKHALIL BRAHMIAMaria Luisa Sapino and Rosaria RossiniPeiman Barnaghi and Azuraliza Abu BakarHammoud Aljoumaa and Dirk SoeffkerHaris GavranovicMatthias MoutonMohammad Reza DaliriFABIO STELLARohit J. KateRuoqian Liu, Yi Lu Murpheyjennifer bavani krishnan kehoe and Rahul SinghGeorge Runger and Mustafa BaydoganMohamed Ghalwash and ZORAN OBRADOVICJieyue LiJosif GrabockaMarco CristaniTomas OlssonTianwei LiuCexus Jean-ChristopheAntonello Rizzi and Antonello RizziZhai Ting TingMing ZhangShanghai Jiao Tong and Zhang ZhongnengZhang Zhang and Dacheng TaoTrevor TianTa Minh ThuyYin Zhou and Kenneth BarnerSuzanne Tamang and Simon Parsonscarlotta orsenigo and carlo vercellisBahaeddin Eravci and Hakan FerhatosmanogluHaobo JiangBingyi KangJing ZhangJon Froehlich and Yi-Chun KoMatthias Klusch and Josenildo SilvaJing YangKomang Sidhi ArthaPhilip Knaute and Nick JonesIvan Mitzev and Nick YounanJuhua Hu and Jian PeiHasari Tosun and John SheppardMd. Abdul AwalAndrey SukhanovCecil SchmidtSebastian Schmitz,Marcin Grzegorzek, Lukas Köping

Benjamin Bustos and Heider Sanchez EnriquezThinh Vuong and Thinh VuongYiorgos AdamopoulosZhao XiaohuiXue BaiBaker AbdalhaqParia Shirani and Mohammad Abdollahi AzgomiManuel Oviedo de la Fuente.JiRinki PakshwarNikita Mishra and Somesh KumarMike JonesDeák Szilárd Nicola RebagliatiIra Assent and Søren ChrestensenMarcela SvarcBankó Zoltányasuko matsubara and Yasushi SakuraiLiu, Yueming and Su, Jianzhonghongxiong ye and QIAN-LI MAHuseyin KayaHezi Halpert and Mark LastPaolo Missier and Tudor MiuJiandong Wang and Peng-Cheng ZOUShahriar Shariat Talkhoonche and Vladimir PavlovicHideo BannaiGerard Medion and Dian GongShengfa Miao and Arno KnobbePenugonda RavikumarNabil Alshurafa and Majid SarrafzadehVladislav MiškovicDavid Providakes and Jason WangWei Ding and Yang MuMeng-Jung Shih and Shou-de LinOSman GunayLu Min and Xiaoru YuanHiba BhamroukhiLaurence ChuPuneet SinghJoaquim Vinicius C AssuncaoB Kalyan Kumar and debarun karJim Austin and Alexander FargusJundong Li and Osmar R. ZaianeBilel Ben aliOya Celiktutan, bulent sankur and Cem SübakanMichael PettigrewKatarzyna Kaczmarek and Olgierd HryniewiczJun ha ParkDaniele Codecasa and Fabio Antonio StellaAnqi Wu and Raghu RaghavendraNur Zuria Haryani and Abdul RazakHamdan Ana Arribas GilZhe Bao and Junpeng BaoAlexio Kotsifakos and Vassilis AthitsosVincent S. TsengRyan Kleck and Gu QuanquanGavin SmithUsue Mori, Jose A. Lozano, Alexander MendiburuMohammed Hasan Al-WeshahSarah Brockhaus and Sonja GrevenFenghuan LiWang YuMayank Mohta and Adit MadanSrinivasulu Reddy and Naga sundaramChiatung Mao and Jia-Dong MAOMaciej LuczakZhiguang Wang and Tim OatesLukas Pfahler and marco stolpeRickey AgarwalYada Zhu and Jingrui HeThanhvinh Vo and Duong Tuan AnhChris CarboneXiaohui HuangNicolas RagotTanmoy MondalGuillem RigaillFatemeh Kaveh-Yazdy and mohammad reza zareAzer KerimovHanan ShteingartXin QiS. MohanavalliEmanuele Ruffaldi and Leonard JohardTomasz PanderPierre Allain and Thomas CorpettiChangcheng XiangWojciech Mioduszewski and Jurek BlaszczynskiZhang DaokunXiangzeng KongAmparo Alonso BetanzosOlivia Maoy and Yixin ChenCarlo GAETAN and Paolo GirardiFarzad Noorian

Greg FanslowXiaojin LiKorkinof, DimitriosYixin Chen and Yujie HeChristina Yassouridisitti laurent and jiaping ZHAOAhmad Al-HasanatArtur Dubrawski, Mathieu Guillame-BertRodica Potolea and Victor IonescuJANETH CAROLINA RENDON AGUIRREJavier PrietoGianniotis, NikosWesley Chen & Pavlos ProtopapasSahar Torkamani and Volker LohwegNajarian, KayvanPengjiang QianHudson Fujikawa and Renato IshiiAntonio GattoneAlexandra M. Carvalho and Susana Vieira and Lucia CruzHanci Lyu and James KwokPeihai ZhaoSergey MilanovZhibo ZhuStefan Kramer and Atif RazaBrijnesh-Johannes JainQian XiaochaoFelix Reinhart and Witali AswolinskiyKwangho HeoCun JiSara Carolina Gouveia Rodrigues and Cláudia AntunesKumar VasimallaJose Alejandro CorderoLiu XiaoCarlos Francisco Soares de SouzaYurong LuoWang, Kaile(Keller)Babak HosseiniDong HanColin O’ReillyAldo GoiaRahim KhanAdam OlinerArvind Balasubramanian and B. PrabhakaranSoon-hwan Kwon, Jong Ho LeeChen Yunndre Rodrigo Sanches and Nina Sumiko Tomita HirataClaudio Piciarelli and Gian Luca ForestiWei T. Yue Michael Botsch and Josef A. NossekBingyu SunSun, Fu-ShingBabak AmiriXing ChunXiao and Du Xutao, Tsinghua UniversityElloumi Samir , Sondess BentekayaLi ShijinErik Learned-Miller, Marwan A. MattarChiranjib Bhattacharya,Karthik KNicandro Cruz RamirezJiankui Guo Fudan UniversityBin Z Zhang IBMYi-Dong Shen and Zhiyong ShenGeorgios Evangelidis, Leonidas KaramitopoulosHendrik PurwinsJignesh M. Patel and Michael MorseGert Van Dijck and Marc Van HulleChao Hui Lee and Vincent TsengLinh Tran (Boeing)Hugo Alonso Vilares Monteiro and Joaquim Fernando Pinto da CostaDavid MinnenTsuyoshi MikamiQiang Yang and Sinno PanPaolo TormeneHui Ding and Peter ScheuermannRonaldo Cristiano PratiChristian Gruber and Bernhard SickSilvia ChiappaAnkur JainMaria Cristina Ferreira de Oliveira and Aretha Barbosa AlencarFebri AndrianiMyeong-Seon GiPengtao jiaFarid Seifi.Clodoaldo Aparecido de Moraes LimaKonstantinos BlekasJuan PradaBen Fulcher and Nick JonesVictor ShengCedric Frambourg and Ahlame Douzal ChouakriaRakia JAZIRI and Mustapha LebbahDave Marshall and Andrew AubreyOmar TorfasonLe Huu Thanh and Duong Tuan Anh.Laila Fatehy and Mahmoud GabrAlle veenstraNima RiahiMohiuddin Ahmad and md. Abdul AwalHyokyeong Lee and Rahul Singh

Inderjit Dhillon and HYUK CHOAida VallsNarayanan Chatapuram Krishnan and Sethuraman PanchanathanStephan Gunnemann and Thomas SeidlThirumaran Ekambaram and M. Narasimha MurtyChristine Preisach and Lars Schmidt-ThiemeDino Isa and Rajprasad KumarFeibao ZhuoFrans van den BerghKfir GlikXiao YuYingying ZhuRosanna Verde and Antonio BalzanellaPaul BaggenstossKoichi ASAKURA and Wei FanLucia Sacchi and Iyad BatalMorne NeserLuca Chiaravallotivikram deshmukhHarri M.T. SaarikoskiCaio Nogara Andreatta and Neusa GrandoYing XieUlf GrobekathoferChristophe GenoliniCeline RobardetMusa ChemistoWangmeng ZuoPaolo RemagninoSoumi Ray and Tim OatesPierre Ganarski and Francois PetitjeanJoydeep Ghosh and John TourishAlireza-XakerWilfgang nejdl and ERNESTO DIAZ-AVILESRomain Tavenard and Laurent AmsalegHahn-Ming Lee, Christos Faloutsos,Hsing-Kuoh Pao, Ching-Hao (Eric) MaoWoong-Kee Lohbikesh singhMarco Grimaldi, Cesare Furlanello and Giuseppe JurmanChonghui GuoSaeid RashidiYanchang ZhangS R KannanFRANCISCO JAVIER CUBEROS GARCIA-BAQUERODr. SOTIRIOS P. CHATZISMing Luo and Igor G. ZurbenkoYuya SakamotoYosi KellerDavid CorneyAlexander Gray and Nishant MehtaTomoyuki Hiroyasu and Takuma NishiiBlaz Strle and Martin MozinaYasin BakisMrinal Mandal and Cheng LuHendrik Blockeel and Kurt DriessensJonas Haustad and Yongluan ZhouJulien RabinOscar Gerardo Sanchez Siordia and Isaac Martin de DiegoScott Deeann ChenLixia WuTewari, AshutoshRANGEENA T V radhakrishnanZhou ZhouTroy RaederAmit ThombreNhut Ta HoangOisin Mac AodhaAli FarahpoorXingwang ZhaoWu GangTetsuji HidakaWang He NanDavid WestonDr. Huayang (Jason) XieJia-Lien HsuSaeed R. AghabozorgiPaolo GiussaniZhenhui (Jessie) Li and Jiawei HanJeff PattiMuthyala Kartheek,Navneet GoyalMojtaba Najafi, Dr. KangavariPhilippe ESLINGZoe Falomir LlansolaXiaoxu HeCheng Po Man and Wai-Kuen ChamDoina Precup and Jordan FrankLaurens van der MaatenGUSTAVO CACERES CASTELLANOSDeepa KrishnaswamyWang Jialin

Some of the people that

requested the UCR Archive

prior to Summer 2015:

Page 1

Tao Quang BangJeremie MaryKai-Wei Chang and Dan RothJoan Serrà and Josep Lluis ArcosRisa Myers and Risa MyersAnkit GuptaWeng-Keen Wong and Yonglei ZhengCássio M. M. Pereira and Rodrigo Fernandes de MelloSubutai AhmadVisuru Lafi Mohamed RiznyEunice Carrasquinha, Ana Pires, Conceição AmadoHuaming Huang and MehrotraAbdulallah BalameshFeng Zhou and Fernando De la TorreJohannes SchneiderJosé Manuel Benítez Sánchez (and Students)Xin Zhao and Xue LiDaniel Gordon, Danny Hendler, Lior RokachAnirvan Chakraborty and Probal ChaudhuriQing He and DongZhiIoannis Paparrizos and Luis GravanoMarcos Quiles.Chris Grieves and Rida E. Moustafa (Shalash)Ben Marlinmaryam tayefiHuiping Cao and Zhe XieChristian Hundt and Elmar SchömerM. en C. Jorge Ochoa SomuanoGoce Ristanoski, James Bailey and Wei LiuPaul Viola (Yes, as in Viola- Jones)JYH-CHARN (STEVE) LIU and Hao WangTSUJIMOTO Takaaki and Prof. UeharaTomáš Kepic and Gabriela KoskováEva Ceulemans and Joke HeylenAlexis BONDUChe-Jui HsuWang ChaoWong, Weng-Keen and Nick SullivanPavel SeninQian XuFengzhen Tang and Peter TinoQiang WangHesam IzakianWitold PedryczYun Chen and Yang, HuiTanatorn Tanantong and Ekawit NantajeewarawatCauchy LeeChedy RaissiBARRE Anthony and RIU DelphineYang ZHANGFlavio Vinicius Diniz de FigueiredoJussara Marques de AlmeidaDr. J. Figueroa-NazunoVictor Romero-Cano and Juan NietoJianhua ZhaoLeslie S. Smith, Abdulrahman A.Samuel J. Blasiak and Huzefa RangwalaHarm de Vries and Azzopardi, G. (George)Folly Adjogou and Alejandro MuruaDustin Harvey and Todd, Michael D.Kilian WeinbergerDaniel Kohlsdorf and Thad StarnerZengwen MoFaezeh Eshragh and Xin WangTuan Dang and Leland WilkinsonKang Li and Yun Raymond FuAndrés Eduardo Castro OspinaZheng ZhangPeter Fox and Yanning(Yu) ChenJeremiah Deng and Feng ZhouFranz Pernkopf and Nikolaus MutsamAlexey ChernyshevBenjamin AuderJiawei Yuan and Shucheng YuHeidelinde Roeder and Jan FreundToon Calders and Thanh Lam HoangFeixiang GaoNancy Pérez-CastroTracy Hu HAndré Genslerzairul hadiBeau PiccartTomas PfisterJohn Costanzo and Nathan CahillConceição Amado and Diogo Silva Zhang, Zhifei and Qi, Hairong Diego Rivera García.

Tanzy Love and Kyra SinghManuele Bicego and Pietro LovatoVladimer KobayashiTalayeh Razzaghi and Petros XanthopoulosMohit SharmaJuergen Schmidhuber and Klaus GreffHeriberto AvelinoManoj ApteNehal Magdy SaberWebber Chen and Dan StashukHuaihou Chen and Philip ReissSeung-Kyu LeeMarcelo de Azevedo ÁvilaSubhajit Dutta and Anil K. GhoshMAILLARD, BrunoYazhou Ren and Carlotta DomeniconiEithon Cadag and Johan GrahnenDinkar Sitaram and Varun ShenoyChan Syin and Vuong Nhu KhueRodrigo AraujoMohamed KamelBorgwardt Karsten M. and Llinares Lopez FelipeJiang LiyangTimothy Ravasi and Gregorio AlanislobatoJames ZhangHeloisa de Arruda Camargo and Fábio José Justo dos SantosD.Pradeep KumarSteffen Dienst and Stefan KühneChristian Hahn and gudrun stockmannshaoran xuChuanlei ZhangYoungha Hwang and Saul GelfandTetsuya NakamuraDaniel Alejandro Garcia LopezRichard J. PovinelliYi-Dong Shen and Zhiyong ShenEmmanuel ViennetRemi Gaudin nicolas nicoloyannisPhil GrossFrancois PortetQi He , Dr. Kuiyu Chang, Dr. Ee-Peng LimFabio Antonio Pereira ReisDamien TessierCuvelier EtienneMoataz El Ayadi & Mohamed S. KamelHillol KarguptaAndrey UstyuzhaninHussam AlshraidehJohn M. TrenkleBoris Martinez Jimenez & Francisco Herrera FernandezJuan Jose RodriguezZoltan BankoAnjan Goswami & Debashis MondalSimon Kagwi MwangiSimone Fontolan and Alessandro GarghettiLin Zhang and Joe SongZhenzhi Huang and Zhenfeng HeWaleed KadousHao Hu and Qiang YangKay Robbins and Dragana VeljkovicDaniel Pena, Regina Kaiser, Ana Laura BadagianiDaniel Graves and Witold PedryczAnne DentonJulia Hunter and Martin ColleyLucia SacchiBernhard Seeger, Michael Grau.John MaindonaldBalazs TormaNikolaos ChatzisDaniel SmithAbdul Razak, Khairuddin OmarElwin (Yong) leeAlex Smola and Xinhua ZhangRudolf Kruse and Christian MoewesPekka SiirtolaMichael BertholdDavid Bong and James TanZhengzheng (Crystal) Xing and Jian PeiLeticia Arco Garcia and Rafael BelloNuno Constantino Castro and Paulo AzevedNg Kam SweeAntonio IrpinoJong Myoung KoJonas RichiardiHassani, Marwan and Sebastian Schaub Chen, Po-Yu,McCann, Julie, Yu, Weiren Laetitia Chapel Michelle Zhang and Sirajul Salekin

Dhaval Patel, Wynne Hsu and Lee Mong Lee.Ville HautamakiPeter SunehagRichard ClementsHichem Frigui and Walid MISSAOUIKASHIMA ToruTang Ke and Guojie SongJinfu FanRuchira GuhaFan Zhou and Wu YueChen DuanshengCheng wencongXiaoli LiFedor Zhdanov and Vladimir VovkWU Quan-YuanAnuradha KodaliKeith Noah SnavelyHilario Navarro Veguillas and Jesus BousoMyong K. JeongLuca GastaldiDileep GeorgeRoni Khardon and Yuyang WANGAlexandros NanopoulosBeibei Zhang and Rong ChenSile O'ModhrainAmy McGovernYa-Ju Fan and W. Art ChaovalitwongseXiaobin LiSmruti SarangiPeter GrabustsFernando Cela DiazThomas Dyhre Nielsen and Shengtong ZhongZhenzhou ChenWesley Kerr and Paul Cohenlaurent baumesLuis AntunesChih-Chun and Zeeshan SyedRene Vidal and Merve KayaZhaozhong WangDuangmalai Klongdee and Chuleerat JarRakesh BabuBao YubinRayner AlfredJim HowardXiaojun Zeng and Geng LiAntonio NemeMITTAPALLI SAI SUDHEER and V.BhavaniJulien JACQUESAbdulla Al Maruf and Kyoji KawagoeNeuza Filipa Martins Nunes and Hugo GamboaLiang GeGeovanny Giorgana and Paul PloegerUgo Vespier and Arne KoopmanBoaz Nadlerjankim luoMike Dessauer and Sumeet DuaChangxin GaoYIN Hong-shengBaiming MaRamon HuertaPetr VolnyFrancois RheaumeAlexander KolesnikovVishwajeet Singh ThakurTomasz GoreckiKerem MuezzinogluRitwik KumarYANG YuhangAli Shokoufandeh , Terence Tuhinanshu, Ernst PischMaria Titah JatipaningrumAnita Santanna, Nicholas Wickstromseyma ketenciKristian HindbergGodtliebsen FredMary She and Jing WangJustin Bayer and Patrick van der SmagtAmmal Al-AnaziKevin ShenStephan Chalup and Arash JalalianLeandro MinkuMichele Dallachiesa and Themis PalpanasAndreas Brandmaier and von Oertzen, TimoGarcia-Trevino, Edgar and Javier A. BarriaHao Zhu and He ZhongshNipotephat MuangkoteAleksandar PechkovMyat Su Yin

Abhishek Sharmashouyi wang and W. Art ChaovalitwongseHyrum AndersonQian ChenTomas Bartos and Tomas SkopalVitali LoseuAlessandro AntonucciAndrii Cherniak and Vladimir ZadorozhnyJulia Gabler and Stephan SpiegelWasyl MalyjPiotr Sobolewski and Michal WozniakIris Adae, Michael Berthold and Sebastian PeterOmid Geramifard and Xu JianxinNg Wee Keong and NGUYEN HAI LONGLi TaoWillian Zalewsk and Fabiano SilvaNeelanjana Dutta, Sriram Chellappa, Donald Wunsch IIDominique GAYMai Thai Son and Christian BöhmIoannis Kaloskampis and Dave MarshallMYAT SU YINAiping ZhouHoutao Deng and George RungerTianyu Li and Dr. DongMay D Wang and James ChengThomas Rueckstiess and Patrick van der SmagtRebecca Willettkrystal taylorPengfei ZhuBohdan KozarzewskiQuangang LiSun XiangzhiYutao FanDimitri Mavris, Megan A Halsey, Curtis IwataWang WeiMªIsabel Borrajo GarcíaDennis Shasha and Lefteris SoulasAndrew CohenWeng-Keen Wong and Xinze GuanNagesh, KarthikThomas Steinke and Patrick SchäferAntonio PentaKatherine Anderson and Castle, Joseph PBressan, StephaneMichael Böhlen and Mourad KhayatiYi ZhengLijuan ZhongEduardo Gerlein and Martin McGinnityDr.Ing. MAURICIO OROZCO-ALZATEKevin Shi and Layne LoriSharon Goldwater and HAIDER AdnanYouqiang SunKoki Nakatani and Kumiko Tanaka-IshiiUttam Kumar SarkarAalaa Mojahed and Beatriz De La IglesiaProf. L. ZhangHrishav AgarwalJimin WangPiyush KumarQing Xie and Xiangliang ZhangAbhay Harpale, Tianbao Yang and Daniel MarthalerAndrew FinchMona RahimiFred NicollsPaulo Borges and George Darmiton da Cunha CavalcantJohn Lach and Davis BlalockHeggere S. Ranganath and Vineetha BettaiahKeith HendersonSteve Lalley and Herbert Xiang ZhuBehzad MansouriMegha AgrawalZhihua Chen and Wangmeng ZuoDiep Nguyen NgocBATCHELOR AndrewJinwook Kim and Min-Soo Kim.Warat Safinpradeep polisettyHoushang Darabi and anooshiravan sharabianiChen QiangLiying ZHENGSarat Kr. ChettriGraham Taylor and Ethan BuchmanCai QinglinAnuj Srivastava and J. Derek Tucker Ajithkumar WarrierKyle Johnston, and Adrian Peter Serim Park and Gustavo Kunde RohdeLee Seversky and Jack M Fischer

Ahmad, Faraz and Smith, DavidVelislav BatchvarovGautier MartiDimitrios KosmopoulosRana Haber and Adrian PeterShachar Afek Kaufman and Ruth HellerChengzhang Qu and Jianhui ZhaoBasabi ChakrabortyGrzegorz DudekSindhu Ghanta and Jennifer DyYu Fang and Huy Xuan DoAli Raza Syed and Andrew RosenbergAgada Josephricardo garcia rodenasOm Prasad PatriLeonardo Nascimento FerreiraMichal PrílepokKe Yi and LUO,GeBruno Ferraz do Amaral and Elaine SousaEdward GONG and Si Yain-WharChaoyi PangVida Groznik and Aleksander SadikovAbdulhakim Qahtan and Xiangliang ZhangCao Duy TruongOliver ObstYung-Kyun NohWei SongYu SuUwe Aickelin, Eugene Ch'ng, Xinyu FuQuang NguyenOr Zuk, Avishai Wagner, Tom HopeSubhadeep MukhopadhyayWEN GUOpoyraz umut hatipoglu and Cem IyigunYongjie CaiMehdi FaramarziTarhio Jorma and Chhabra TamannaXiaoqing WengSimon ShimYan Gao Gao and Daqing HouMaria Rifqi, Marcin Detyniecki and Xavier RenardFrank Englert and Sebastian KößlerPaul Grantjuan colonnaElias EghoQUANG LE and Chung PhamJitesh KhandelwalKarima DyussekenevaWu, Ruhao and Wang, Bo Yuki Kashiwaba Ade Bailly Abhinav VenkatMichael Hauser and Asok RayTuan-Minh Nguyen and Michael Bromberger Ruako TAKATORI Rahul Singh and Ryan EshlemanLaura Beggel Tom ArodzVictoria Sanchez Mike Sips and Carl Witt Segolene Dessertine-Panhard Zhao XuLovekesh VigAngelo Maggioni e Silva Abdelwaheb Ferchichi and Mohamed Saleh Gouider Wenlin Chen and Yixin ChenGui Zi-Wen and Yuh JyeRomain Tavenard Chandan GautamJinglin PengAhlame Douzal and Jidong YuanChen JingZeda Li and Alan J. IzenmanPeng ZhangHanyuan Zhang and Xuemin TianGeorgios Giantamidis and Stavros TripakiGoutam Chakraborty and Takuya KamiyamaPipiras, Vlada and Stone, Eric Ping LiHerbert KruitboschAdam Oliner, Jacob Leverich, Tian Chen, Tom LaGatta Yulin SergeyHuan LiuVadim VaginNikolaos Passalis Graham Mueller

Some of the people that

requested the UCR Archive

prior to Summer 2015:

Page 2

Appendix A: Sharpshooter Plots

function plot_texas_sharpshooter() % Compute a Texas Sharpshooter plot of DTW over Euclidean Distance. See SDM 2011 paper% Batista, Wang and Keogh (2011) A Complexity-Invariant Distance Measure for Time Series. SDM 2011 [ texas_names = data_names; % Note that the order of texas_names and texas_values must be the same.texas_values = 1-error_rates; % Note that here we convert error to accuacy, by subtracting from 1 expected_accuracy_gain = texas_values(:,2)./texas_values(:,1);actual_accuracy_gain = texas_values(:,3)./texas_values(:,1); plot(expected_accuracy_gain,actual_accuracy_gain,'r.'); % Produce plot just so we can get Xlim and YlimXaxis = get(gca,'XLim');Yaxis = get(gca,'YLim'); clfhold on;axis square; patch([Xaxis(1) 1 1 Xaxis(1)],[Yaxis(1) Yaxis(1) 1 1 ],[0.9843 0.8471 0.5765]); % Bottom left quadrantpatch([1 Xaxis(2) Xaxis(2) 1],[1 1 Yaxis(2) Yaxis(2) ],[0.9843 0.8471 0.5765]); % Top right quadrant plot(expected_accuracy_gain,actual_accuracy_gain,'r.'); xlabel('Expected Accuracy Gain');ylabel('Actual Accuracy Gain'); for i = 1: length(texas_values(:,1)) %text(expected_accuracy_gain(i),actual_accuracy_gain(i),int2str(i))end for i = 1: length(texas_values(:,1)) text(expected_accuracy_gain(i),actual_accuracy_gain(i),texas_names(i,:),'rotation',+30)end end function names = data_names() names =...['Plane ''Car ''Synthetic Control ''Gun-Point ''CBF ''Face (all) ''OSU Leaf ''Swedish Leaf ''50Words ''Trace ''Two Patterns ''Wafer ''Face (four) ''Lightning-2 ''Lightning-7 ''ECG ''Adiac ''Yoga ''Fish (readme) ''Beef ''Coffee ''OliveOil ''CinC_ECG_torso ''ChlorineConcentration ''DiatomSizeReduction ''ECGFiveDays ''FacesUCR ''Haptics ''InlineSkate ''ItalyPowerDemand ''MALLAT ''MedicalImages ''MoteStrain ''SonyAIBORobot SurfaceII ''SonyAIBORobot Surface ''StarLightCurves ''Symbols ''TwoLeadECG ''WordsSynonyms ''Cricket_X ''Cricket_Y ''Cricket_Z ''uWaveGestureLibrary_X ''uWaveGestureLibrary_Y ''uWaveGestureLibrary_Z ''Non-Invasive Fetal ECG Thorax1 ''Non-Invasive Fetal ECG Thorax2 ''InsectWingbeatSound ''ECG5000 ''ArrowHead ''BeetleFly ''BirdChicken ''Ham ''Herring ''PhalangesOutlinesCorrect ''ProximalPhalanxOutlineAgeGroup ''ProximalPhalanxOutlineCorrect ''ProximalPhalanxTW ''ToeSegmentation1 ''ToeSegmentation2 ''DistalPhalanxOutlineAgeGroup ''DistalPhalanxOutlineCorrect ''DistalPhalanxTW ''Earthquakes ''MiddlePhalanxOutlineAgeGroup ''MiddlePhalanxOutlineCorrect ''MiddlePhalanxTW ''ShapeletSim ''Wine ''WordSynonyms ''Computers ''ElectricDevices ''FordA ''FordB ''HandOutlines ''LargeKitchenAppliances ''Meat ''Phoneme ''RefrigerationDevices ''ScreenType ''ShapesAll ''SmallKitchenAppliances ''Strawberry ''UWaveGestureLibraryAll ''Worms ''WormsTwoClass ']; end function e_rates = error_rates() e_rates =... [ 0.0380 0 0 0.2670 0.2830 0.2330 0.1200 0 0.0170 0.0870 0.0400 0.0870 0.1480 0 0.0040 0.2860 0.0380 0.1920 0.4790 0.2450 0.3880 0.2110 0.1760 0.1540 0.3690 0.2330 0.2420 0.2400 0.0100 0.0100 0.0900 0.0020 0.0015 0.0050 0 0.0050 0.2160 0.1250 0.1140 0.2460 0.1000 0.1310 0.4250 0.2000 0.2880 0.1200 0.1400 0.1200 0.3890 0.3970 0.3910 0.1700 0.1700 0.1550 0.2170 0.2000 0.1540 0.3330 0.5000 0.3330 0 0 0 0.1330 0.1000 0.1330 0.1030 0.0700 0.0700 0.3500 0.3660 0.3500 0.0650 0.0630 0.0650 0.2030 0.1740 0.2030 0.2310 0.0750 0.0880 0.6300 0.4710 0.5880 0.6580 0.5800 0.6130 0.0450 0.0450 0.0450 0.0860 0.0180 0.0860 0.3160 0.2600 0.2530 0.1210 0.2000 0.1340 0.1410 0.1480 0.1410 0.3050 0.1000 0.3050 0.1510 0.0930 0.0950 0.1000 0.0400 0.0620 0.2530 0.0870 0.1320 0.3820 0.2620 0.2520 0.4230 0.2000 0.2280 0.4330 0.2410 0.2380 0.4130 0.2260 0.2540 0.2610 0.2490 0.2270 0.3380 0.2750 0.3010 0.3500 0.2950 0.3220 0.1710 0.1770 0.1850 0.1200 0.1220 0.1290 0.4380 0.4820 0.4220 0.0750 0.0580 0.0750 0.2000 0.0830 0.2000 0.2500 0.1500 0.3000 0.4500 0.1500 0.3000 0.4000 0.1470 0.4000 0.4840 0.4840 0.4690 0.2390 0.2110 0.2390 0.2150 0.2200 0.2150 0.1920 0.1820 0.2100 0.2920 0.3020 0.2630 0.3200 0.3500 0.2500 0.1920 0.1110 0.0920 0.2180 0.3170 0.2280 0.2480 0.2500 0.2320 0.2730 0.3600 0.2720 0.3260 0.2730 0.2580 0.2600 0.4740 0.2530 0.2470 0.2160 0.3180 0.4390 0.4090 0.4190 0.4610 0.4500 0.3000 0.3890 0 0.3890 0.3820 0.2620 0.2520 0.4240 0.2480 0.3800 0.4500 0.1820 0.3760 0.3410 0.3350 0.3410 0.4420 0.4140 0.4140 0.1990 0.1000 0.1970 0.5070 0.1730 0.2050 0.0670 0 0.0670 0.8910 0.7850 0.7730 0.6050 0.3330 0.5600 0.6400 0.4560 0.5890 0.2480 0.1880 0.1980 0.6590 0.2830 0.3280 0.0620 0.0780 0.0620 0.0520 0.0280 0.0340 0.6350 0.5840 0.5860 0.4140 0.4160 0.4140];end

Here is the code we used to produce the sharpshooter plots.

The Password

• As noted above. My one regret about creating the UCR archive is that some researchers see improving accuracy on it as sufficient task to warrant a publication. I am not convinced that this should be the case (unless the improvements are very significant, or the technique is so novel/interesting it might be of independent interest).

• However, the archive is in a very contrived format. In many cases, taking a real world dataset, and putting it into this format, is a much harder problem than classification itself!

• Bing Hu explains this nicely in the introduction to her paper [2], I think it should be required reading for anyone working in this area.

• So, the password is the three redacted words from this sentence “Every item that we ******* ## @@@@@@@ belongs to exactly one of our well-defined classes”, after you remove the two spaces.

• The sentence is on the first page of [2].