Using Rhythm Awareness in Long-Term Activity Recognition

Kristof Van Laerhoven, David Kilian, and Bernt SchieleTU Darmstadt, Germany

{kristof, kilian, schiele}@mis.tu-darmstadt.de

Abstract

This paper reports on research where users’ activitiesare logged for extended periods by wrist-worn sensors.These devices operated for up to 27 consecutive days, dayand night, while logging features from motion, light, andtemperature. This data, labeled via 24-hour self-recallannotation, is explored for occurrences of daily activi-ties. An evaluation shows that using a model of the users’rhythms can improve recognition of daily activities signifi-cantly within the logged data, compared to models that ex-clusively use the sensor data for activity recognition.

1. Introduction

This paper presents work towards a vision where peoplewear a wristwatch-type logging unit for extended periods oftime, without taking it off when going to sleep: the device iswith them at all times. Unlike commercial actigraph units,the recordings are fine-grained and descriptive enough tocharacterize particular motion patterns and postures, whichlink to recurring activities such as ”riding a bike”, ”takinga nap”, or ”using a computer”.

Long-term recording of wearable sensor data has thus farbeen done almost exclusively in the medical field. The re-sources required for achieving this have been typically highfor both users and medical staff. Having detailed and con-tinuous activity data spanning several weeks to months, al-lows a wide variety of applications. Numerous scenarioswould benefit from this type of in vivo monitoring, such asdetecting straining activities for post-operative surgical pa-tients [2], or correlating mood swings of psychiatry patientswith particular activities (e.g., ”taking the stairs” versus”sitting in sofa”) [4].

Logging such detailed activity data for weeks to monthsis challenging however, and the number of the body loca-tions where a sensor device can be worn for 24/7 comfort-ably is limited. We operate under the assumption that theuser’s dominant wrist is an optimal location, familiar fromwristwatches, provided the sensor unit is small and light

Figure 1. Users upload and label activity data daily froma continuously logging wrist-worn sensor. Rhythmicmodels are built to improve recognition in future data.

enough. Mobile devices such as phones are good candi-dates as well, but were not selected as they are not alwaysas closely worn on the body and can easily change orienta-tion and location.

This paper proposes to build a rhythm model from theuser’s daily activities. When for instance the wrist’s mo-tions and postures do not lead to a conclusive recognition ofthe activity ”giving presentation”, the user’s habit of givinga lecture every Tuesday afternoon might improve this. Thisrhythm model is trained and evaluated by weeks to monthsof continuous user-specific data – Figure 1 shows how datais uploaded and annotated daily – and contributes to auto-matic generation of activity annotations.

Our hypothesis is that these rhythms will result in an in-crease in recognition accuracy for actions that occur regu-larly at a certain circadian frequency, compared to a clas-sifier that is based upon just the sensor data (from inertialsensors, light sensors, and temperature). We will conduct avery similar study as in [3], but with models that are builtfrom our on-body inertial motion and posture data, insteadof computer activity.

2. Long-Term Data Collection

A first prerequisite in this work is having a light-weightyet powerful wearable sensor that is able to record motion,posture, light, and temperature for days in a row. But alsohaving a way of obtaining annotations for the daily activitiesover long term data proved to be challenging.

2.1. Wrist Sensor

Compared to commercial actigraph units, most notably[1], we do not only log the levels of activity more frequentlyand in three axes, but record posture information as well.This substantial increase in information results in a decreasein battery-lifetime however. To relax this trade-off, twotypes of inertial sensors, accelerometers and tilt switches,are alternated to avoid storing redundant data and to saveenergy (see [4] for details). Features of the sensor data arestored on a microSD card of up to 4 Gigabytes. The designfiles, includes electronic schematics for the printed circuitboard, the software for the microcontroller and client-sidesoftware for hardware-interaction, are freely available fordownload from the project site1.

For the experiments described in this paper, we limitedthe unit to log only the means per axis and the combinedvariance at approximately 5Hz. When the accelerometersare turned off, the tilt switch states are recorded by takingthe first read tilt switch states, as well as the accumulatedhamming distance over 200 tilt switch readings. Uploading24 hours of data from the SD card via USB takes about 30seconds. The data is then converted from binary to a commaseparated text file for further analysis (with one day worthof data taking up approximately 16 Mb in binary format or42 Mb in text).

A substantial amount of time and effort went into mak-ing the wrist-worn unit as robust as possible, while keepingit small, light, and comfortable to wear. Several iterationswere required to achieve a system that we could confidentlyhand out to users to wear for periods of several days to sev-eral weeks, without supervision by the research team.

2.2. Daily Self-Recall Annotation

To assist the users in annotating their own recordings, asoftware package was developed, which performs two basicfunctions: 1) It automatically initiates a connection via USBto the wrist-worn sensing unit, sets the time, and downloadsthe recorded data, and 2) it visualizes these data on-screen.Users can then select segments on the visualization to anno-tate the data from recalling what happened during the last 24hours, or by observing the data.

1http://porcupine2.sourceforge.net

Figure 2. Two of the wrist-worn sensing units used inthis work, without straps, the left showing an epoxiedversion, while the right shows the populated board.

The advantage of such annotation is that it does not de-mand much effort on behalf of the test subjects: most par-ticipants use maximally 15 minutes, including the time todownload the data to the PC and to export to comma sep-arated data files. A disadvantage is that the annotation isgenerally coarse and depends heavily on the user’s recall,however this effect might be minimal since most daily ac-tivities in the experiment lasted for at least 15 minutes.

3. Rhythm Models from Sensor Data

The rhythm model’s basic element is a bin, which rep-resents a region of time that characterizes the sensor valuesseen in that particular segment, as well as the annotationsthat have been associated with those values. The activitiesare also explicitly modeled by accumulating the times thisparticular annotation started and stopped per bin, as well asthe durations. This representation is close to that in [3], butdoes not require the inference of transitions between activ-ities, or labels for those activities, as these have been pro-vided in the data sets. The most basic rhythm model wehave implemented is that of a day: for all days, we com-bine sensor data and annotations in bins of 5 minutes. Eachbin therefore represents the data and annotations that aretypically present within the bin’s time region (for instancebetween 14:20 and 14:25), from all days in the data set.

The model’s bins contain the mean and standard devia-tions of the feature samples observed from all days in thecorresponding 5-minute time segments. These feature vec-tors are stored as abstractions of all data over all days thatwere sampled during the 5-minute segment of the bin, andare normalized when used for recognition. Through the ac-tivities’ stored start-, stop-, and duration times in units ofthese bins, the rhythm model contains three probability dis-tributions for each activity that is present in the recordings.The start and stop distributions are combined to construct amodel which allows to look up per 5-minute bin what thelikelihood is of an activity occurring, the durations are thenused to smooth this start-stop model so that a better densityestimation is reached with just a few weeks of data. This

time of day

Activity 24/02/2008 − 23/03/2008 (27 days), Bin size: 5 mins

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

25/0226/0227/0228/0229/0201/0302/0303/0304/0305/0306/0307/0308/0309/0310/0311/0312/0313/0314/0315/0316/0317/0318/0319/0320/0321/0322/03

in car

0 240

13

star

t

0 240

8

stop

0 20 400

5

leng

th

breakfast

0 240

14

0 240

6

0 10 200

5

10

lunch

0 240

3

0 240

3

0 10 200

5

10

sleeping

0 240

4

0 240

7

50 100 1500

5

10

forest walk

0 240

2

0 240

2

0 20 400

5

10

workshop

0 240

2

0 240

6

0 500

5

10

shower

0 240

8

0 240

8

2 3 40

5

10

dinner

0 240

3

0 240

2

0 20 400

5

10

sauna

0 240

1

0 240

2

10 150

1

2

Activ

ity R

hyth

m M

odel

Cale

ndar

Day

sAc

tivity

Rhy

thm

Mod

elCa

lend

ar D

ays

Activ

ity R

hyth

m M

odel

Cale

ndar

Day

s

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 240

0.1

0.2

0.3

0.4

0.5

0.6

0.7

in carbreakfastlunchsleepingforest walkworkshopshowerdinnersauna

Figure 3. Top: Annotations of subject 1 during 27 days,the legend below shows the activities with their start,stop and duration histograms. Bottom: a one-dayrhythm model from these annotations and times.

means that for typical short activities, such as for examplethe activity ”showering” in Figure 3 between 4:30 and 5:00,the model remains short as well, whereas activities with alonger duration are smoothened more, as for instance theworkshop activity.

4. Method and Experiment Data

The prime reason for building the rhythm models in thispaper is the improvement they might bring to the detectionof daily activities in the recorded sensor data. This was in-vestigated on two data sets of four consecutive weeks each.

The evaluation is done using K-fold cross-validation pertest subject, where the entire data set is partitioned into Kdays, and classification is performed on a single day, whilethe other days are used to train that classifier. Four-weekdata sets were taken from two people: a 61-year old con-

struction foreman with a rhythmic working schedule (seeFigure 3), and a 24-year undergrad student with a highlyirregular one. Activities included ”(a) driving car”, ”(b)having breakfast”, ”(c) having lunch”, ”(d) sleeping”, ”(e)forest walk”, ”(f) workshop”, ”(g) taking shower”, ”(h)having dinner”, ”(j) in sauna”, ”(k) riding bicycle”, ”(l)playing saxophone”, ”(m) working on computer”, and ”(n)watching TV”.

In order to test the classification without the rhythmmodel, we chose to use a simple distance-weigthed K Near-est Neighbor classifier by representing each annotation inthe data set with the 5-minute segments that were associ-ated with it, for all days. There is much room for improve-ment in this classification scheme, but since the objectivehere is to investigate the impact after the addition of therhythm model, we will leave more elaborate classifiers asfuture work.

For the combination, we explored several methods andfound that the best results were obtained when the activityrhythm model did the classification after the closest distancefrom the KNN classifier reached a preset threshold. As thetest subjects forgot to annotate activities at certain occasions(e.g., see the missing night segments in Figure 3), a back-ground class was not used.

5. Classification Results

The results from the KNN classifier (see Table 1 for pre-cision and recall per activity) show an overall accuracy of82-84%, but this is heavily skewed because of the fact thatthe ”sleeping” activity is easy to recognize, and occurs al-most one-third of the time per day. The important thingto note here is that activities such as ”(h) dinner”, or ”(b)breakfast” do not get recognized at all from the sensor dataalone for subject 2, and many others perform poorly. Theactivity ”(d) sleeping” is recognized almost perfectly, whilemany other activities such as ”(h) dinner” and ”(b) break-fast” are rating poorly for subject 1. Subject 2’s activitiesperform particularly poorly, with only ”(d) sleeping” and”(m) working on the computer” doing well.

Results of the classifier when combined with the rhythmmodel shows a slight overall increase, again because ofwell-performing dominant activities such as ”(d) sleeping”,but the improvement is more visible when looking at the re-sults per activity. Especially for structured eating activities(”(b) breakfast”, ”(c) lunch”, and ”(h) dinner”), the in-crease is significant with a 10 to 20 percent increase in bothrecall and precision. In total, especially the recall for thecombined method has increased significantly (on average a10% increase for subject 1, and a 5% increase for subject 2).The average precision performed only 2% better for subject1, and suffered a small drop for subject 2’s activities, be-cause of an increased confusion between both ”(l) playing

KNN on data only - Subject 1:a b c d e f g h j total

R 0.76 0.42 0.35 0.99 0.55 0.60 0.74 0.26 0.43 0.57P 0.74 0.49 0.37 0.98 0.55 0.55 0.73 0.30 0.64 0.60

KNN on data only - Subject 2:a b c d g h k l m n total

R 0.19 0 0.19 0.98 0.35 0 0.39 0.53 0.86 0.46 0.40P 0.31 0 0.24 0.97 0.49 0 0.48 0.58 0.80 0.56 0.44

.

KNN & Rhythm model - Subject 1:a b c d e f g h j total

R 0.81 0.60 0.40 0.99 0.65 0.47 0.85 0.50 0.76 0.67P 0.79 0.69 0.48 0.99 0.54 0.59 0.77 0.37 0.49 0.63

KNN & Rhythm model - Subject 2:a b c d g h k l m n total

R 0.17 0.44 0.31 0.98 0.37 0 0.36 0.53 0.83 0.5 0.45P 0.14 0.27 0.34 0.98 0.49 0 0.22 0.24 0.83 0.65 0.42

Table 1. Recall (R) and Precision (P) values for KNN only,and combined with the rhythm model, for subjects 1 and2 (see section 4 for activity descriptions a-n).

saxophone” and ”(k) riding bicycle” which were hard toclassify via the data method already – other activities suchas ”(b) breakfast” have benefitted significantly. A rhythmmodel incorporating the day of the week, or having morediscriminative features than the mean and variance, wouldlikely increase performance here.

Some of the low precision and recall measures encoun-tered in this experiment are most likely the result of thecoarse-grained annotation of our technique and the com-plex nature of the activities that were annotated to the sen-sor data. First of all, many activities such as ”breakfast”,”lunch”, and ”dinner” are very hard to recognize with datafrom just the low-level light and motion sensors. Most ofthese in our experiment are however very time dependent,so it is not surprising that the usage of the rhythm modelimproves recognition for these activities. With more data,of several months instead of several weeks as was the casein our experiments, activities that take place on certain daysof the week (such as ”sauna”) can be expected to be recog-nized better as well.

Finally, a simple classifier such as KNN works for someactivities already very well on its own, even among severalother activities. The inclusion of a rhythm model has notshown noticeable effects on such activities and did not harmthe other activities’ recognition in this case, but it is possiblethat this might change when better features are incorporatedin the wrist sensor, combined with activities that are notrhythmic at all.

Obvious issues stemming from the annotations are thelow resolution at which the users marked the sensor data,which currently makes it impossible to insert short butmeaningful activities in the model. Next versions of theself-recall software will include functionality to zoom inand out of the visualizations in the annotation tool, so thatthis might become feasible, perhaps with inclusion of but-ton presses on the sensor unit to make it easier to pinpointevents on the 24 hour data plots.

6. Conclusions

We have presented initial studies around a long-term an-notation method, in which users annotate their own data.Using a small-scale and customized hardware platformwhich allows users to record basic features from wrist-wornsensors, and a software tool to annotate the data visually, wehave undertaken a study where two test subjects indepen-dently annotated their own activities, once per day, in theirhomes without supervision from the researchers. A rhythmmodel that aims to capture the user’s behavioral rhythms inday-to-day activities is proposed as a way to cope with ac-tivities that are hard to recognize from just the sensor data.

The data sets from the subjects, each holding one monthof continuous data sampled at approximately 5Hz and an-notated daily by the user, showed that the rhythm modelcan improve the activity recognition, especially for the pre-cisions of activities that belong strongly to these rhythmichabits, such as ”having breakfast”, ”relaxing in the sauna”,and ”watching tv”.

Future work will have to verify whether this improve-ment holds true for other scenarios activities that do not con-form to any daily or weekly rhythms, and whether rhythmmodels built from longer periods beyond one month willhave similar effects. Current improvements to the hard-ware allow for more descriptive features of the accelerom-eter data, and faster transfer of the data via USB. Finally,more test subjects will complement those that already wearthe sensing units, so that future experiments can work ondata from for a wider demography and for several monthsinstead of just one.

7. Acknowledgments

This research has been funded in part by the MobVisproject (EU IST-2002-2.3.4.1). The authors would also liketo thank the test subjects, who were kind enough to wearour hardware for extended periods of time.

References

[1] Actiwatch. http://www.minimitter.com/Products/Actiwatch.[2] O. Aziz, B. P. L. Lo, G.-Z. Yang, R. King, and A. Darzi.

Pervasive body sensor network: An approach to monitoringthe post-operative surgical patient. In International Workshopon Wearable and Implantable Body Sensor Networks (BSN2006), pages 13–18. IEEE Computer Society, 2006.

[3] J. Begole, J. C. Tang, and R. Hill. Rhythm modeling, visual-izations and applications. In UIST, pages 11–20. ACM, 2003.

[4] K. V. Laerhoven, H.-W. Gellersen, and Y. G. Malliaris. Long-term activity monitoring with a wearable sensor node. In BodySensor Networks (BSN 2006), pages 171–174. IEEE Com-puter Society, 2006.