the use of apps to promote energy saving: a study of smart … · 2019. 8. 10. · original article...

ORIGINAL ARTICLE

The use of apps to promote energy saving: a study of smartmeter–related feedback in the Netherlands

Daphne Geelen & Ruth Mugge & Sacha Silvester &

Annemieke Bulters

Received: 23 May 2017 /Accepted: 13 November 2018 /Published online: 28 February 2019

Abstract Feedback systems with direct feedback haveshown to be effective in stimulating households to changetheir energy consumption levels. This research is one of thefirst to explore the use of apps to influence householdenergy use. Compared to dedicated in-home displays,smartphone/tablet apps provide a low-cost and simpledesign solution for making energy feedback available.This research consisted of three studies conducted withdifferent samples within a selection of households where asmart meter was installed as part of the smart meter imple-mentation program in theNetherlands. First, for a period of16 months, electricity and gas consumption levels weremeasured for a large sample of households (n = 519) di-vided into an application user group and a reference group.Second, questionnaires (n = 270) provided insight in how

people used the applications and to what extent the appli-cations increased households’ insight in their energy con-sumption and stimulated behavior changes. Third, inter-views (n = 12) were held to obtain more in-depth insight.In the sample with measured energy consumption, we didnot find a significant reduction in electricity and gas con-sumption during this research. Yet in the questionnaires,the application users reported more energy awareness andindicated to have made more investments and changes intheir behavior than the reference group. Most app usersstarted using the first app they found and did not explorethe other options. The interview results indicate that, afteran initial learning period, the app was used to monitor theelectricity and gas consumption levels, rather than to lowerthem. In line with other research into feedback, the inter-view results suggest that the apps could be more effectivewith information that is more actionable and meaningfulwith respect to one’s own specific situation and goals forthe household. Further exploration is recommended withrespect to how the design of such apps can encourage awide audience not only to monitor their consumption, butalso guide them in taking action to change their consump-tion levels.

Keywords Apps . Smart energymeters . Directfeedback .Energysaving .Behaviorchange .Households

Introduction

A quarter of the final energy consumption in the EuropeanUnion is accountable to households (European Union

Energy Efficiency (2019) 12:1635–1660https://doi.org/10.1007/s12053-019-09777-z

D. Geelen :A. BultersEnexis B.V., ‘s-Hertogenbosch, Netherlands

D. Geelene-mail: [email protected]

A. Bulterse-mail: [email protected]

R. Mugge (*) : S. SilvesterFaculty of Industrial Design Engineering, Delft University ofTechnology, Delft, Netherlandse-mail: [email protected]

S. Silvestere-mail: [email protected]

# The Author(s) 2019

http://orcid.org/0000-0003-1091-1652

http://crossmark.crossref.org/dialog/?doi=10.1007/s12053-019-09777-z&domain=pdf

2016). Households are therefore an important target groupfor policy measures aimed at energy efficiency. In theEnergy Efficiency Directive communications, theEuropean Commission states its goal to Bensure majorenergy savings for consumers^ and its aims to Bempowerenergy consumers to better manage consumption.This includes easy and free access to data on con-sumption through individual metering.^ (EuropeanCommission 2017). To make consumption informa-tion available to households, EU member states haveto equip households with smart meters, whenever itis cost-effective (European Union 2012).

Why is it so important to make consumption informa-tion available via smart meters? A Bsmart meter^ is anelectronic energy meter that is remotely readable and canthus communicate consumption data continuously andautomatically. Smart meters are introduced to provideoperational gains for suppliers and distribution networkoperators, such as automated and accurate billing, frauddetection, and differentiated tariffs, as well as to provideconsumers with more information about their energy con-sumption. The underlying premise for unlocking con-sumption data to consumers is that when consumers re-ceive energy feedback information, they will gain insightin their energy consumption pattern andwill be able to takemeasures to lower their consumption. These measuresinclude changing behavior routines and investment inenergy efficient technology, such as insulation and efficientappliances. In light of the ongoing energy transition, smartmeters are also expected to play an important role infacilitating products and services that enable house-holds to (automatically) adjust their consumptionpatterns to the availability of renewable electricalenergy and to contribute to the balancing of supplyand demand in the grid (Geelen et al. 2013;Giordano et al. 2011; Kobus et al. 2015a).

Energy feedback information can be provided in vari-ousways and it is crucial that the data from the smart meteris communicated to consumers in a way that is useful andmeaningful to them. The EU directives do not prescribehow the households should get insight into their energyconsumption, and member states decide upon their ownpolicies in this respect. For example, in the UK, eachhousehold in which a smart meter is installed is providedby default with a display that visualizes the energy con-sumption levels in real-time (Ofgem and DECC 2011). Inthe Netherlands, the policy is that households receive a bi-monthly overview of their electricity consumption fromtheir energy supplier (Van Elburg 2014). For more direct

and detailed feedback, households can acquire one of thefeedback systems that are available on the market. Amarket analysis looking into the development and uptakeof feedback systems in the Netherlands concluded thatthere is a need for accessible and cheap systems to appealto households with little interest or limited means to pur-chase a feedback system (Van Elburg 2014). The studyalso emphasized that it is important that feedback is simpleto use for those consumers who are not Internet savvy ornaturally interested in monitoring their energy consump-tion. A review of the products and services in the Dutchmarket (www.energieverbruiksmanagers.nl) confirms theneed for cheap and accessible feedback systems. Themajority of products and services on offer aresmartphone/tablet software applications (apps) that areavailable for free. Whether these are effective for bringingabout energy savings for a wide audience is still in ques-tion. There has been limited research on how energyfeedback via apps can help households to save energy overan extended period. This study contributes to the knowl-edge on energy feedback systems by exploring howhouse-holds engage with a simple and low-cost feedback systemin the form of an app in a natural setting and the effects ofsuch a system on their household energy consumption.

Literature review

Previous research has shown that providing feedbackinformation about a household’s energy consumptioncan lead to energy savings, see, e.g., the reviews byAbrahamse et al. (2005); Darby (2006); Fischer(2008); and Ehrhardt-Martinez et al. (2010). The re-views indicate that different ways of providing the feed-back information as well as the duration of feedbackintervention produce different achieved savings. Energysavings tend to be lower with longer intervention pe-riods and in larger scale trials (Ehrhardt-Martinez et al.2010). For example, energy savings from in-home dis-plays are expected in the range of 3–5% when imple-mented in large-scale trials, rather than 6–10% found insmall-scale studies (McKerracher and Torriti 2013). Thereviews also show that differences in the achieved sav-ings are also due in part to differences in the design offeedback systems and in how people engage with thefeedback. The question now is therefore not so muchwhether feedback works, but how it works and whichdesign factors influence the effectiveness of feedback(Karlin et al. 2015; Van Dam et al. 2010).

1636 Energy Efficiency (2019) 12:1635–1660

http://www.energieverbruiksmanagers.nl

In the design of feedback systems, we have to con-sider the ways in which users engage with the feedback,how the feedback is presented, and which information isprovided. One of the key factors for design andevaluation is the immediacy with which the feedbackis available. Often, the distinction is made between whatDarby (2006) identified as Bdirect feedback^ versusBindirect feedback.^ Direct feedback is directly avail-able to the user—on demand—and presents real-timeinformation. Indirect feedback becomes available onlywith a delay that may range from a day to a year.Overall, direct feedback has shown to be more effectiveand the more immediate the feedback information ismade available, the more effective it is (Abrahamseet al. 2005; Darby 2006; Fischer 2008; Ehrhardt-Martinez et al. 2010; Karlin et al. 2015).

The direct availability of feedback information doesnot imply that users frequently consult the information,which is a separate and important factor. Systems withhigher use frequencies have shown to be more effective(Kobus et al. 2015b). To achieve a high-use frequen-cy, the feedback information should be easily acces-sible and provide information that is relevant for theuser. Furthermore, additional functionality of thefeedback system is suggested to motivate frequentuse, such as the combination of feedback with ther-mostat control, weather information, or dynamictariffs (Kobus et al. 2015b).

From the literature, it is clear that the design of thesystem should facilitate users’ engagement with thefeedback system a normal part of daily life. This impliesthat the feedback system should support positive dialogsbetween household members about energy manage-ment, and that the feedback system is accessible andattractive to all household members (Van Dam et al.2012; Hargreaves et al. 2010).

To use the feedback for energy saving, the presenta-tion of the information has to be understandable, rele-vant, and actionable for households (Van Dam 2013;Hargreaves et al. 2010; Geelen et al. 2013). For themeasurement unit in which the feedback is expressed,users prefer costs (€, $) over energy units (kWh, m3)(Karjalainen 2011). A note here is that Karlin et al.(2015) did not find significant moderating effects fordifferent measurement units, which suggests that otherdesign factors may be more influential. Showing num-bers is not directly necessary for feedback to be under-standable and relevant; color-based information hasbeen found to be easily understandable and appreciated

(Bonino et al. 2012) and ambient displays are also ableto convey information that is actionable for users (Darby2006). Ambient displays do not necessarily show num-bers but give an impression about energy consumptionbased on, for example, changing colors, a flashing light,or a sound.

The granularity of the feedback, i.e., the level ofdetail represented, is also considered a factor that influ-ences the effectiveness of feedback. Usually, the distinc-tion is made between appliance-specific and aggregatedconsumption data. However, there is no consensus as toits effects. The results from studies by Karlin et al.(2015) and Kobus et al. (2015b) suggest thatappliance-specific information is not always useful to auser, because the feedback may not tell users how toreduce consumption (Karlin et al. 2015) or because theaccessibility of the information does not stimulate fre-quent use of the feedback (Kobus et al. 2015b).

Comparative information can help users to interprettheir consumption data to create meaningful informa-tion. Historic consumption enables users to understandwhat their energy consumption pattern looks like(Anderson and White 2009; Fischer 2008) and to com-pare their current patterns (per day, week, month, year)with their past consumption. This can provide users witha personal norm as to what a normal or desirable con-sumption pattern is (Karlin et al. 2015). Another form ofcomparison is normative or peer-usage comparison, inwhich a household’s consumption is benchmarked withthat of other (similar) households. Although this form ofcomparison is often appreciated by users, making goodcomparisons is also complicated by differences amongindividual households (Stankovic et al. 2016). Historicconsumption is considered to be more effective (Fischer2008), though when compared with goal-based compar-isons, i.e., goal-setting, Karlin et al. (2015) did not findsignificant effects for normative and historical compar-isons. They suggest that goal-based comparison is moreeffective because it motivates users to focus their atten-tion on the actions required to meet a goal that isrelevant to themselves. In a similar vein, it is recom-mended to make interfaces goal-driven, i.e., produc-ing actionable feedback that helps users to under-stand the extent to which a given goal is beingapproached and what actions may be required tomeet the goal (Geelen et al. 2013).

The medium through which the feedback is providedlargely defines the accessibility of the feedback. In thisrespect, in-home displays are considered most

Energy Efficiency (2019) 12:1635–1660 1637

promising, because they make feedback directly visiblein the household. While in-home displays have shownpromising results (e.g., Kobus et al. 2015b; Van Damet al. 2010), we should be wary of overconfidence inthese devices (Nilsson et al. 2014; Buchanan et al. 2015)because it is not the fact that people have a display, butthe overall design of the display that influences the waypeople understand and use the feedback. Furthermore,these devices are quite expensive1 and thus it is impor-tant to investigate whether low-cost feedback systemscan also provide households with the needed insights toreduce their energy consumption.

This study contributes to the literature with insight inthe effectiveness of a feedback system that uses asmartphone or tablet application as a medium to providedirect feedback. Little is known about the use and ef-fectiveness of feedback systems in the form of an app,but an app provides a low-cost alternative to an in-homedisplay and is therefore more attainable for a large userbase than an in-home display.

Method

This research examined households in which a distribu-tion network operator replaced their old energy meterwith a smart meter. In a trial, the distribution networkoperator offered its customers a feedback system on thepremise that this feedback system could help the house-holds gain better insight in their energy consumptionand save energy. This context provided a unique oppor-tunity to observe the effects of a feedback system in anatural setting. To be able to see the effects of thefeedback system, we made sure to collect data from agroup of households with feedback systems as well asfrom a reference group that did not have them. In thischapter, we will discuss the characteristics of the feed-back system and apps, the research design, and partici-pant recruitment.

The feedback system and design of the apps

The apps that were tested in this research make use ofelectricity and gas consumption data that was measured

by the newly installed smart meter in the participatinghouseholds. The smart meter has a so-called P1-port, towhich a P1-reader is connected. The P1-reader makesthe measurement data from the smart meter available forthe apps via the household’s Wi-Fi-network.

The households that were offered a feedback systemfor this research could make use of two smartphone/tablet apps, a desktop (Windows) application, and anonline application. The online application was princi-pally used to connect the smartphone/tablet app anddesktop application to the P1-reader. See Table 1 foran overview of the characteristics of the available energyfeedback. The majority of the application users used asmartphone/tablet app (84%).

The smartphone/tablet apps were developed by dif-ferent organizations. The researchers could not influ-ence the design of the apps because the research projectwas initiated by a Dutch distribution network operatorthat is responsible for large parts of the electricity net-work in the Netherlands. The network operator’s goalwas to allow other organizations to develop their ownapps based on their organization’s preferences. As aresult, the app features differed on several aspects, suchas functionality and graphic design.

The apps were developed as a low-cost and low-maintenance solution with a short time for development.The developers therefore used design features that arecommonly used for energy feedback.With respect to theguidelines described in the literature review, these appsincluded direct feedback aggregated to the total house-hold consumption and historic comparisons. Both de-signs were built up in layers. The Bhome^ screen fo-cused on the actual consumption levels (direct feed-back). By clicking on items on this screen or navigatingthe menu items, one could access more detailed infor-mation, such as historic consumption (indirect feed-back). The two apps differed in Blook and feel^ and inthe ways they presented energy consumption informa-tion. App A (Fig. 1) reports energy consumption withnumbers and per counter of the smart meter, whereasApp B (Fig. 2) reports energy consumption graphicallyin a dial, complemented with numbers. App B processesthe raw consumption data more than App A, in order toprovide meaning to the data and facilitate interpretationof the data graphically. The extra features of App Binclude a comparison between current consumptionadding up over the day with the total consumption ofthe previous day (Fig. 2, left), comparison of historicdaily consumption totals per week or month with the

1 In the Netherlands, feedback systems with in-home displays are sold inthe market for at least €100, sometimes in combination with a service feeof €18 to €55 per year. See www.energieverbruiksmanagers.nl(Accessed: 5 February 2018).


http://www.energieverbruiksmanagers.nl

Tab

le1

Characteristicsof

thefeedback

system

pertestgroup(including

referencegroup)

Smartm

eter

Reference

group

Applicationusergroup

App

AApp

BDesktop

application

Onlineapplication

Yes

Yes

Yes

Indirectfeedback

Bi-monthly

overview

from

supplier*

Bi-monthly

overview

from

supplier*

Bi-monthly

overview

from

supplier*

Directfeedback

functio

ns/-

inform

ation

–Current

consum

ptionfrom

and

deliv

erytothegrid.Shownin

abargraphandnumbers(W

per10

s,m

3perhour)

Current

consum

ptionfrom

thegrid.

Show

non

adialandnumbers(kW

and

m3)

Current

consum

ptionfrom

and

deliv

eryto

thegrid.S

hownin

abargraphandnumbers(W

&kW

hper10

s,m

3perhour)

Thistool

mustb

eused

toenable

theapplications

toreceivedata

from

theP1-

reader.

Historicconsum

ptionperhour,

dayandmonth

percounter.

Presentedin

numbers(kWh

andm

3).

Totalconsumptionof

todayandyesterday.

Presentedin

bargraphandnumbers(€)

Historicconsum

ptionperhour,

dayandmonth

percounter.

Presented

inbargraphs

and

numbers(kWhandm

3).

Current

consum

ptionfrom

and

deliv

eryto

thegrid.S

hownina

bargraphandnumbers(W

&kW

hper10

s,m

3perhour)

Historicconsum

ptionperhour,day

and

month

percounter.Presented

inabar

graph(€),combinedwith

consum

ption

oftheprevious

day/week/month.

Current

meter

readings

(kWhand

m3)

Tipsforenergy

saving,accessedviaalin

kto

awebsite.

Dow

nloadof

historic

consum

ptionpercounter.

Setting

amaxim

umconsum

ptiongoal(in

€)with

analertw

henthemaxim

umis

reached.

(Novisualizationof

historicdata)

Tobe

obtained

via

–App

Store

GooglePlay

(indicated

inmanual)

App

Store

GooglePlay

Feedbacksystem

website

(indicated

inmanual)

Webpage

ofrouter

(indicated

inmanual,required

forinstallatio

nof

theP1-reader)

Toinstallo

n–

Smartphone

Tablet

Smartphone

Tablet

Com

puter(W

indowsO.S.only)

Not

applicable.T

hisisawebsite.

*Householdswith

asm

artm

eter

receiveabi-m

onthly

overview

oftheirelectricity

andgasconsum

ptionby

defaultfrom

theirenergy

provider


consumption levels of the week or month before (Fig. 2,right), setting of an alarm for when a certain level ofconsumption costs for the day is reached (differentscreen), and tips for energy saving (directing to awebsite).

It is important to note that both the application usergroup and the reference group received bi-monthly con-sumption overviews (indirect feedback) throughout thestudy (see Table 1). By default, households in theNetherlands with smart meters each receive a bi-monthly

Fig. 1 Main screens of app A.Left: Current consumption ofelectricity and gas in kWh andm3,depicted in a bar graph. Right:Historical consumption per hour,day, or month for each counter ofthe smart meter

Fig. 2 Main screens of app B.Left: Current consumption ofelectricity and gas in kWh and m3

depicted as a speedometer andcomparison between today’s andyesterday’s consumption inEuros. Right: Historicalconsumption in Euros per day(pink) compared with days of theprevious weeks (gray)


consumption overview. This overview is provided by thehousehold’s energy provider. Because this bi-monthlyoverview is provided to all participants in the research,we have included questions in the questionnaires andinterviews about it in order to evaluate the effect of theapps in comparison to the bi-monthly overview.

Research setup

The conceptual model for this research (Fig. 3) followsthe reasoning underpinning the policy for smart meterimplementation, namely that the use of the feedbacksystem results in increased insight in one’s energy con-sumption and thereby enables a household to changedaily energy-related behavior and/or make changes tothe home that result in reduced energy consumption.The continued use of the feedback system enables theuser to increase the learning (insight) about the house-hold’s energy consumption and perform further changesin daily behavior or investments. This conceptual modelis used as a basis for the data collection.

This research is quasi-experimental, given that wewere bound to the possibilities and decisions made bythe network operator for the research methods and par-ticipant recruitment participants per method. The datacollection was organized as three studies with differentthough complementary goals, methods, and samples.

The three studies include the following:

1. Measurement of electricity and gas consumption toinvestigate, with a large sample, whether the avail-ability of the feedback system has an effect onenergy consumption levels. The consumption levelsof households with a feedback system were com-pared with those of a reference group.

2. Questionnaires to gain quantitative insight in theeffects of the feedback system on energy awareness,behavior change, and evaluation of the feedback, aswell as demographic characteristics. To be able todetermine effects, the questionnaires were executedwith households with a feedback system, as well aswith households in a reference group. To evaluatedifferences between the app user groups, the ques-tionnaire included questions about app use and appcharacteristics.

3. Interviews with app users to gain insight into howpeople used the apps. The interviews were used tocomplement the quantitative insights from thequestionnaires.

In the following sections, we describe the researchprocedure and recruitment process for each study.

Procedure

Measurement of electricity and gas consumption

For a period of 16 months, the electricity and gas con-sumption per day were measured for households thatconsented to have their meters read by the distributionnetwork operator. The energy consumption data werequantitatively analyzed, comparing the differences be-tween the application user group and the referencegroup. Before the analysis, the dataset was cleaned byremoving double entries and correcting erroneous datapoints due to errors in registration of the data.Additionally, cases with extreme values (M ± 2SD)wereremoved from the sample because these may relate toincorrect meter readings, and we did not want theseextreme values to influence the overall results. This

Use of thefeedback system

Behaviourchange

(Daily behaviour& Investments)

Insight in energy

consumption

Changes in consumption

levels(kWh, m3)

Fig. 3 Conceptual model


meant that all cases with values above 20-kWh dailyelectricity consumption were removed from the set ofelectricity consumption data (4% of the total cases wasremoved) and all cases with values above 8 m3 daily gasconsumption were removed from the set of gas con-sumption data (3% for application user group and refer-ence group combined). No cases were removed due toextreme low values (M-2SD) because this correspondedto negative consumption (− 1 kWh and − 4 m3 respec-tively) and these were not present in the sample.Additionally, the yearly consumption was collected foreach household for the year before the intervention(2013) and the years the intervention ended (2014 and2015).

Questionnaires about application use, insightin consumption, and energy-saving behavior

Questionnaires were developed for telephone question-naires of approximately 10 min. We opted for telephonequestionnaires because of the higher response rates forthis method compared to online questionnaires. Thequestionnaires were held up to 5 months (T1) and upto 11 months (T2) after installation of the smart meter.They were conducted over a period of 3 to 4 weeks. Thequestionnaire for T1 focused on the short-term effects ofthe feedback system and the questionnaire for T2 on theeffects in the longer term. The second questionnaire alsoaddressed the default bi-monthly overview. For the sec-ond questionnaire, only the people who had respondedto the questionnaire for T1 were contacted.

The main topics for respondents in both the applica-tion user group and the reference group were as follows:

– Perceived insight in one’s household energy con-sumption with the feedback system

– Extent to which the used feedback system helps tosave energy

– Effects on energy-related behavior, i.e., changes indaily activities and implementation of energy effi-ciency measures

In case of the reference group, the feedback systemrefers to the smart meter and the bi-monthly overview.

The application users were additionally asked about:

– Which application was (most) used– How often they used the application

In the Appendix Table 5, a list of the questions andanswer options is provided. Five-point scales were usedas much as possible for the scale variables. Other ques-tions were categorical or included an open answer op-tion. With the survey results, we aimed to compare thegroups’ perceived insight in energy consumption andbehavior changes.

Interviews

To gain deeper insight into how people used the apps,semi-structured face-to-face interviews with a durationof about 1 h were held. These interviews were used tocomplement and explain the findings from the energydata measurements and questionnaires. The interviewstopics included the following: how people used the appin their daily lives, how the app helped households gaininsight into their energy consumption, how people dealtwith energy-saving measures for their home, and theextent to which the other household members wereinvolved in the use of the app and energy saving.

The interviews were held in two rounds, within6 weeks after the first and second questionnaire respec-tively. The interviews were fully transcribed, coded, andanalyzed to explain results from the questionnaires.

Because of the qualitative nature of this study, a smallsample size could suffice, as long as we reached datasaturation. When there are repeatedly only one or twonew insights (or codes) for new interviews, one hasreached Bdata saturation^ (Guest et al. 2006). Guestand colleagues (Guest et al. 2006) concluded that, witha group of relatively heterogeneous individuals, 12 in-terviews should suffice to reach data saturation.Hagaman and Wutich (2017) suggest that 12 to 16interviews are sufficient for a focused topic and hetero-geneous group.

Participant recruitment

For the recruitment of research participants, we adheredto the standard process applied by the distribution net-work operator so that the research conditions would beas close as possible to the default process of smart meterinstallation and evaluation. We therefore first made aselection of addresses for installation of a smart meteraccording to the standard process, namely about 5000addresses. We only included single-family houses insmall cities and villages in the Netherlands, becausethe majority of Dutch dwellings are single-family


houses and these are thus an important target group forenergy-saving measures. A portion of these addresseswas designated for recruitment to the application usergroup (approx. 3500) and a portion for the referencegroup (approx. 1500). The selection for the applicationuser group was larger because we had to take intoaccount a response rate for applying to the P1-readeras well as for participation in the research. For theselection of the reference group, we only had to considerthe response rate for participation in the research.

All households were sent a letter announcing theinstallation of the smart meter in their home accordingto the standard procedure of the distribution networkoperator. After installation of the smart meter, they weresent a smart meter information package by mail. Forhouseholds assigned to the application user group, thispackage also included a flyer with the offer to apply forthe feedback system, and they were sent follow-upletters reminding them to apply for the system. Thefeedback system had to be requested via an onlineregistration form. After registering, P1-readers were sentto the homes of the applicants, who could install themthemselves with guidance from an installation manual.The applicants could also download and install the appsthemselves and connect the apps to the P1-reader assoon as the latter was connected to the smart meter.The households were free to choose an app.

The installation of smart meters took place over aperiod of several months according to the availability ofinstallers and householders. In the end, 1428 householdsapplied for the feedback system and could thus beapproached for energy consumption measurement,questionnaires, and interviews as part of the applicationuser group. For the reference group, we could reach outto approximately 1500 households, provided that theinstallation of the meter had been successful.

Recruitment for energy consumption measurement

With respect to energy consumption measurement,households had to consent explicitly to daily meterreading by the distribution network operator. To obtainthis permission, the households with a feedback systemwere sent an e-mail. For the reference group, a randomselection of households was called by phone with therequest to sign up via an online form. The response was18% for the households with feedback system and 17%for the reference group (see also Table 2). The gas andelectricity consumption of 519 households could be

measured for a period of 16 months in 2014 and 2015(application user group n = 264; reference group n =255).

Recruitment for the questionnaires

For the questionnaires at T1, addresses were selectedrandomly from the application user group and the refer-ence group. To avoid a bias in our sample, we did notprioritize approaching the households who participatedin the meter readings study for participation in thequestionnaire study. At T2, only households that hadparticipated at T1 were approached, and in the applica-tion user group only those who were using an applica-tion at T1. This resulted in complete questionnaires atT2 of n = 119 for the application user group and n = 151for the reference group. There was a small number ofhouseholds for which both questionnaire results andenergy consumption measurements were available atT2, namely n = 42 for the application user group andn = 55 for the reference group.

Recruitment for the interviews

Participants for the interviews were selected from thequestionnaire respondents who had a feedback systemand indicated willingness to participate in an interview.A total of 12 interviews with application users wereincluded in the analysis. Six of the interviews wereconducted at T1, and six at T2. Because the interviewsetup was the same at T1 and T2, and no comparisonswere made across time, we treated the set of interviewsas one. After the 9th interview, we noted only incidentalnew insights (codes) in our analysis (see Fig. 4), whichassured us that we had reached saturation. Thus, thesample of 12 interviews provided sufficient insights tounderstand the application users’ experiences.

Results

In this section, we describe the composition of thesample to control for differences in demographic vari-ables between the application user group and the refer-ence group. Then we present the results from the anal-ysis of consumption levels, followed by the results fromthe questionnaires concerning the use of the applica-tions, effects on the respondent’s insight in energy con-sumption, and behavior changes in the household.


Lastly, the results from the interviews are described,which provide insights that help explain the effectsfound in the analysis of the consumption data and ques-tionnaires. The number of respondents per study issummarized in Table 2.

Demographics

The demographic differences between the applica-tion user group and the reference group wereassessed with independent t tests and chi-squaretests, based on the responses in the question-naires. Significant differences were found onlyfor gender. The reference group consisted of few-er male respondents than did the application usergroup (70% vs. 90%; χ2(1, n = 224) = 12,564,p < 0.001). It is noteworthy that there are moremale than female respondents. Given that werequested the person in the household that hadbeen most involved with the smart meter andfeedback system to respond to the questionnaire,it appears that in multiple person households, themen were most often using the system (assumingthat most of the households with two or morehousehold members include male-female couples).See Table 3 for the averages of the sample.Because there are no significant differences onthe other variables and given that the majorityof respondents in both groups is male, we assumefor this study that the application user group andthe reference group are comparable with respectto the variables related to the use of the feedback,insight in energy consumption, and behaviorchange.2

Effects on energy consumption

To find out whether energy saving took place inthe application user group, the consumption levelsof the households were normalized based on theconsumption the year before the intervention (af-ter/before*100). This normalization allowed for acomparison of the situation before and after theintervention an independent t test with the con-sumption levels as dependent variable and thegroup as independent variable was performed. Nosignificant differences between the two groupswere found, for electricity (t(476) = 0.1228;p > 0.05) and gas consumption (t(511) = − 0.353;p > 0.05). Thus, no significant difference in energyconsumption was found between the applicationuser group and the reference group.3

Figures 5 and 6 show the gas and electricityconsumption for a period of 16 months during thestudy, aggregated to an average per month of thedaily consumption. Note that the electricity con-sumption for the application user group is struc-turally higher than for the reference group, notonly during the study but also in the years pre-ceding the study. The difference over the totalconsumption during the study period is 7.7% andsignificant (independent t test; t(461) = 2.053;p < 0.05). The graphs in Figs. 5 and 6 illustratethat no energy saving took place in the applicationuser group, because if energy saving had occurredin that group the consumption levels of the appli-cation users would have decreased over time incomparison to the reference group’s levels.

2 A comparison between users of App A vs. users of App B did notshow significant differences.

3 This analysis was also conducted for the electricity and gas consump-tion of the respondents to the questionnaire. No significant differencesbetween the application user group and the reference group for elec-tricity (t(80) = − 1.155; p > 0.05) and gas consumption (t(76) = − 0.564;p > 0.05).

Table 2 Number of participants per study. Response rate between parentheses

Study group Meter read-out Questionnaires Interviews

T1 T2

Application users 264 (18%) 303 (34%) 119 (42%) 12

Reference group 255 (17%) 454 (17%) 151 (74%) –

Total 519 757 270 12


Results from questionnaires

Analysis of the questionnaire responses providedinsight into the differences between the applicationuser group and the reference group, between theusers of app A and app B, as well as changes overthe 6-month period between T1 and T2. We de-scribe first the results concerning the use of thefeedback system, second the effects on the respon-dent’s insight in energy consumption, and third theeffects of the feedback system on the energy-related behavior of the households. The meanscores for the variables discussed below are pre-sented in Table 4. The distribution of the re-sponses for the variables is included in theAppendix Table 5.

Use of the feedback system

As described above, the households had to applyfor the feedback system and received it for free.

The results of the questionnaires showed that themain reasons for applying were interest in gaininginsight in one’s energy consumption or in savingenergy (64%). Most households who used thefeedback system with one of the applications atT1 still used it at T2 (70%). Twenty-nine percentof the people had used more than one application(56 out of 196 respondents at T1). For theserespondents, we asked which application they usedmost. For questions related to application use, wefocused on the most used application, because itcan be expected that this application has mostinfluence on the application user’s behavior. Themajority of the application users mainly used onlyone of the smartphone apps, namely 55% usedApp A and 29% used App B at T1. The choicefor the most used app was based on it being thefirst app people installed; this was either because itwas the only one that—they thought—was avail-able or for its functionality (respectively 23%,26%, and 18% of the responses at T1).

At T2, 64 of 97 respondents (66%) were stillusing the initial app, while 4 had switched toanother app and 29 had stopped using an applica-tion. The most mentioned reason that people hadstopped using the feedback system between T1and T2 was that they did not perceive added value(7 out of 19 answers). Other reasons mentionedmore than once included that the participant didnot make the effort to reconnect the P1-reader tothe internet, to reinstall apps, or look up pass-words (e.g., after changing phones) (3 out of 19),that he or she did not have time (2 out of 19), andthat the system no longer functioned (2 out of 19).

Table 3 Overview of demographics

Variable Value

Mean age of respondent 55

% male 79%

Mean amount of household members 2.65 persons

Mean floor surface 141.7 m2

Mean construction year of the house 1976

Mean gas usage (2013) 1737 m3

Mean electricity usage (2013) 3843 kWh

83

98

112

122125

129135

139 140 141 142 142

0

20

40

60

80

100

120

140

160

1 2 3 4 5 6 7 8 9 10 11 12

Cum

ula

tive n

um

ber o

f codes

Interviews

Fig. 4 The cumulative number ofcodes per interview


At T1, the applications were used several times aweek on average. The use frequency dropped signifi-cantly the longer the app had been in use. Between T1and T2, the use frequency went from an average score ofBseveral times a week^ to an average score of Bonce aweek^ (F(1,67) = 49,325; p < 0.001). A Mann-WhitneyU test revealed no significant difference in the usefrequency of App A and App B.

Effects of feedback system and time on insight in energyconsumption

A repeated measures ANOVA was performed withthe expected insight as dependent variable, thegroups (application user group vs. referencegroup) as between-subjects factor, and time (T1–T2) as within-subject factor. Application users

0

1

2

3

4

5

6

7

8

9

10

Sept Oct Nov Dec Jan Feb March April May June July Aug Sept Oct Nov Dec

Ave

ra

ge

da

ily g

as c

on

su

mp

tion

(m

3/d

ay)

Time (months)

Application users Reference

Fig. 5 Average daily gas consumption per month

0

1

2

3

4

5

6

7

8

9

10

11

12

13

Sept Oct Nov Dec Jan Feb March April May June July Aug Sept Oct Nov Dec

Average d

aily e

lectr

icity c

onsum

ption (

kW

h/d

ay)

Time (months)

Application users Reference

Fig. 6 Average daily electricity consumption per month


reported expecting more insight into their house-holds’ energy consumption levels than did thereference group (F(1,195) = 20,269); p < 0.01).For both groups, the expected insight diminishedbetween T1 and T2 (F(1,195) = 6278; p < 0.05).

The application users also reported the extent towhich the application helped them to save energy.There was a significant difference between T1 andT2 (Wilcoxon signed-rank test, p < 0.05). At T2,the scores were lower than at T1. Applicationusers appreciated the application more than thebi-monthly overview in helping them to save en-ergy (Wilcoxon signed-rank test, p < 0.05). AMann-Whitney U test did not show a significantdifference between App A and App B in terms oftheir tendencies to help save energy.

Effects of feedback system and time on the household’sbehavior

At T1 and T2, the questionnaire respondents wereasked about the extent to which they had adjustedtheir daily behavior. A repeated measures ANOVAwas performed with behavior change as dependentvariable, the groups (application user group vs.reference group) as between-subjects factor, andtime (T1–T2) as within-subject factor. The appli-cation users reported more behavior change thandid the reference group at both T1 and T2(F(1,222) = 11,704; p < 0.001). Furthermore, bothgroups indicated an increase in behavior changebetween the first and the second questionnaire(F(1,222) = 21,584; p < 0.001).

Table 4 Mean scores of the questionnaire results for the differentgroups. See Appendix Table 5 for more details about the possiblescores for each topic. The superscripts in the 4th and 5th columns(a, b, and c) indicate significant differences between test results,

e.g., the superscript a for the results on insight in energy consump-tion indicates a significant difference between the application usergroup and the reference group

Topic Group n T1 M (SD) T2 M (SD)

Use frequency Application user group 68 6.19 (1,22)a 4.79 (1,70)a

Reference group N.A.

Total N.A.

Insight in energy consumption Application user group 94 2.52 (0,65)ac 2.32 (0,75)bc

Reference group 103 2.01 (0,77)ac 1.96 (0,85)bc

Total 197 2.25 (0,76)c 2.13 (0,82)c

Help with energy-saving application Application user group 68 3.34 (0,96)a 2.76 (0,88)a

Reference group N.A.

Total N.A.

Help with energy-saving bi-monthly overview Application user group 60 N.A. 2.58 (0,85)a

Reference group 89 N.A. 2.45 (1,14)a

Total 149 N.A. 2.50 (1,03)

Behavior change–daily behavior Application user group 97 1.90 (1,03)ac 2.29 (1,04)bc


Total 224 1.70 (0,98)c 2.04 (1,09)c

Behavior change –changes to home Application user group 97 No: 86 (89%) No: 66 (68%)

Yes: 11 (11%)a Yes: 33 (32%)ab

Reference group 127 No: 117 (92%) No: 112 (88%)

Yes: 10 (8%) Yes: 15 (12%)b

Total 224 No: 203 (91%) No: 178 (79%)

Yes: 21 (9%) Yes: 46 (21%)

Behavior change–intention to take measures Application user group 97 2.52 (1,05)ac 2.09 (1,03)bc


Total 224 2.24 (1,09)c 1.84 (0,97)c


A repeated measures ANOVA with behaviorchange as the dependent variable, time as thewithin-subjects factor, and application (App Aand App B) as the between-subjects factor indicat-ed a main effect of time (F(1,79) = 11,077;p < 0.01), but did not reveal a significant effectof the apps. Thus, for both apps, more behaviorchange was reported at T2, but we did not findsupport for the possibility that one app performedbetter than the other.

To investigate whether the application use fre-quency was positively related to changes in dailybehavior, we performed a correlation analysis. Theextent of daily behavior change correlated signifi-cantly with use frequency (at T1: Spearman’s rho =0.279, p < 0.01; at T2: Spearman’s rho = 0.473,p < 0.01). The same applies for the intention tomake changes to the house (at T1: Spearman’srho = 0.320, p < 0.01; at T2: Spearman’s rho =0.365, p < 0.01). Thus, the more frequently peopleused the app, the more likely they were to reportbehavior change.

The respondents were also asked whetherthey had actually replaced appliances or madeadjustments to their home (answering with yesor no). At T1, 9% of all respondents reported tohave taken measures. A chi-square test did notshow a significant difference between the applica-tion users and the reference group at T1. For theresponse at T2, a chi-square test showed that theapplication user group had taken significantly moremeasures than the reference group, namely 32%vs. 12% (χ2(1, n = 224) = 13,680 p < 0.01).Furthermore, at T2, more application users report-ed to have taken measures than at T1 (McNemar’stest, p < 0.05).

When comparing the users of App A with thoseof App B, chi-square tests did not show significanteffects between the two apps at T1 or at T2. Moreusers of App A reported energy-saving measures atT2 than at T1 (McNemar’s test, p < 0.05). Forusers of App B, this effect was marginally signif-icant (p = 0.065).

In an open question, the respondents statedwhat adjustments they had made. The most men-tioned measures at T2 were as follows: changingthe lights, changing insulation, and buying energyefficient appliances, at respectively 47%, 14%, and12%.

We also asked about the extent to which re-spondents intended to make adjustments to theirhome in the future. A repeated measures ANOVAwith intention as dependent variable, the groups(application users and reference group) asbetween-subjects factor, and time (T1–T2) aswithin-subject factor was performed. There weremain effects for group and time. The intention tomake adjustments was higher in the applicationusers group than in the reference group(F(1,222) = 17,792; p < 0.001) and the intentiondropped in both groups between T1 and T2(F(1,222) = 25,468; p < 0.001).

To assess differences in the effects of App A vs.App B, a repeated measures ANOVA was per-formed, with the extent to which adjustments wereintended as the dependent variable, time as thewithin-subjects factor, and the app (App A andApp B) as the between-subjects factor. Thisshowed a main effect of time (F(1,79) = 13,835;p < 0.01) but no significant effect between theapps. Thus, for both apps, higher intention tomake adjustments was reported at T2, but thespecific app used caused no effect.

The decrease in reported intention may be ex-plained by the fact that households took measuresbetween T1 and T2, and as a result had lessintention to make more adjustments. Alternatively,it may be that over time households had clearerideas about whether they would implement theinitially intended changes. If they had become lessinclined to, their intention would lessen.

When respondents indicated an intention tomake adjustments, they were asked to clarifywhat they intended to do. Figure 7 shows thedistribution of responses over the mentioned mea-sures. Most mentioned were insulating (25%),installing solar panels (20%), and replacing lights(16%). It is remarkable that 18% of the respon-dents stated that they did not have a concrete ideayet.

Interview results

The analysis of the consumption data did notdemonstrate energy saving in the group of appli-cation users compared to the reference group. Thequestionnaire results, on the other hand, indicatedthat the application users perceived more insight


into their consumption and reported more energy-saving behavior. The results from the interviewsprovided more insight into this apparent contradic-tion by shedding light on how the applicationusers utilized the feedback system. The resultspresented here are based on statements by multiplepersons.

Use of the feedback system

The interview results indicate that the applicationusers used the applications initially to gain insightinto their consumption and then to monitor theirconsumption levels over time. The decrease in usefrequency between T1 and T2 can be explained bythis transition. In the beginning, the users woulduse the app more frequently to learn how theirenergy patterns were built up. Some of the respon-dents actively checked the consumption of specificappliances by switching them on and off or bytrying to explain the total consumption (BWhatappliances are currently on?^). After a while, whenthey had sufficient insight in their consumptionpatterns (Bknow how it works^), they mainly usedthe app for monitoring purposes, which was doneon a less frequent basis. When deviations from theusual known pattern were seen, a user would lookfor an explanation and, if possible and desired,take measures. Different respondents discoveredin this way that, for example, the oven was notfunctioning well, that the door of a built-in refrig-erator did not close anymore, or that the heatingelement of the dishwasher was not functioning.

With respect to the main functions of the apps,interviewees explained the different uses of thecurrent consumption screen and the historical con-sumption screen. The current consumption infor-mation was used to understand where and whenenergy was consumed, how much energy-specificappliances used, and to find out where energy waswasted. The historic consumption was used tomonitor consumption over time, to explain chang-es, and to check for deviations from the usualpattern. In general, historic consumption per day/week/month was perceived to be more relevantthan the current consumption. We expect that thecomparison over time made the information moremeaningful to the application users, because theycould interpret the differences between high andlow consumption during a specific day, for exam-ple by knowing that the washing machine wasused or that the heating was off because no onewas at home. The historic information was alsoused to monitor and control the time of day thatappliances are used. Someone who had conscious-ly taken energy efficiency measures would look atthe historic consumption graphs to see if the mea-sures were actually resulting in lower consumptionlevels compared to before. In this respect, some ofthe respondents stated that they were waiting tocomplete a year of historical consumption data sothat they could compare particular months withthose of the year before.

The development of habits around the use ofthe application seems to be related to the usef r equen cy. One u s e r c he ck ed t h e da i l y

25%

20%

18%

16%

10%

5%

5%

1%

1%

0% 5% 10% 15% 20% 25% 30%

Insulate

Install PV solar panels

No concrete plans

Replace lights

Buy more energy efficient appliances

Adjust heating system

Change behaviour

Install wood stove

Install water saving tap/shower head

Fig. 7 Intended adjustments on T2 as a percentage of the total number of responses (154) for a particular measure (n = 136)


consumption pattern before going to bed andwould regularly discuss with his wife how todecrease their consumption. Others would consultthe app as part of their regular Bplaying^ withtheir smartphone and checking of social mediaapps . O the r s wou ld open the app Bj u s tsometimes,^ or consult the app at a moment whenthey Bjust wondered^ how much the current energyconsumption was.

In each case, the interviewee was the only per-son in the household who had installed and usedthe application. This person, who could be calledthe Benergy manager,^ was often the person in thehousehold who stimulated and executed theenergy-saving activities in the household and whodealt with the contract with an energy provider.Most respondents explained that they discussedinsights from the app with their partner (10 of 11who lived with a partner). The involvement ofhousehold members ranged from being more orless actively involved in maintaining or findingnew energy-saving behavior, to leaving energymanagement to the Bmanager,^ and following his/her suggestions. During the interviews, we metone household where both partners were active asBenergy managers.^ One of them used the app andreviewed bi-monthly overviews, while the otherdealt with the energy provider—Bshe knows thepassword^—and had noted down the meter read-ings on a weekly basis before they had a smartmeter.

In the questionnaires, we found that applicationusers perceived improved insight in their energyconsumption. Given the additional insights fromthe interviews, we can say that this improvedinsight would only have translated to an increasedawareness in the whole household if the energymanager shared insights from the app with house-hold members and they were also (becoming) in-volved in energy-saving activities.

Insight in energy consumption and help for energysaving

From the interviews, we learned that the applica-tions can lead to insight into household energyconsumption expressed on several levels: (1)Insight into the amount of energy the householdconsumes and the pattern of consumption, (2)

Insight into the consumption of specific appliancesand how those figure into overall consumption, (3)Insight into how one’s consumption compares toother households or to what is Bnormal,^ (4)Insight into possibilities for energy saving, namelyby knowing which appliances, technologies, orbehaviors contribute to the desired energy con-sumption levels. While the first two levels werementioned by most respondents, the third andfourth were mentioned by only a few of theinterviewees.

There were various opinions on the extent towhich the apps actually helped to save energy.Some people were content with the provided in-sight and stated that it helped them to save energy,while others required more concrete, detailed in-formation. These latter were not sufficiently moti-vated by the available information because it didnot provide them with the tools to take action.This may also explain why 18% of the respon-dents with intentions to take energy-saving mea-sures were not able to mention concrete plans (seeFig. 7). One respondent, for example, said thehousehold had already saved a lot (insulation, ef-ficient appliances, energy-saving routines) and nowneeded more detailed information in order to opti-mize appliance use. A common remark whenasked about the use of the feedback was that theywondered how to utilize the information the appprovided. They wanted more actionable insightsthat are applicable to their own situations.

Changes to the household’s energy-related behavior

Some interviewees stated that they had becomemore aware of their consumption and thus werelooking for—and taking advantage of—energy-sav-ing opportunities for their homes and daily rou-tines. For example, one respondent had structurallylowered the set temperature of the central heatingsystem since he had installed the feedback system.Others indicated that they had replaced light bulbsor purchased more efficient appliances like dish-washers, washing machines, and refrigerators.Several respondents suggested that the feedbackitself did not cause the energy-saving action, butthat it helped to inform or incentivize decisionmaking. Intentions thus appear to be reinforcedby the feedback system.


The interviewees also shed light on factors thatmay have limited the effects of the feedback onenergy saving. One important reason for little be-havior change that the interviewees mentioned wasthat they already had energy-saving habits and hadalready made several investments to increase (ormaintain) the household’s energy efficiency.Another reason was that respondents did not wantto lessen their general comfort, even though theyexpected that there were possibilities to save moreenergy.

In cases in which the feedback information wasused to evaluate the current situation and thehousehold was satisfied with the consumptionlevels, e.g., because it was below the average forsimilar households or within one’s budget—no ac-tion was taken. Additionally, energy efficiency in-vestments or changes in behavior were not likelywhen the household did not have concrete ideasabout what action(s) to take. In order to actuallytake action, the household must take the extra stepof making the effort to find out how to save moreenergy. A user’s motivation to save energy shouldbe high enough to take this extra step.

The quest ionnai res indica ted that moreenergy-saving measures were taken in the longterm at T2. This may be explained from theinterview results, which suggest that householdsfirst must take action to understand what theycan and want to change, and then make changesat moments that suit them. Two respondentsstated that they had asked for advice on the bestoption for their specific situation. One of themexplained that he was getting advice and quota-tions for a new air conditioner, but had notdecided yet. Furthermore, respondents would of-ten wait for the right time to make adjustments,for example because an appliance had not yetreached its end of life or because the householdcould not yet afford a certain investment. Thiswas mentioned in connection to changing lightbulbs and appliances such as refrigerators andwashing machines, and insulation.

Discussion

As noted in the introduction of this article, apremise for the promotion of energy feedback

systems is that people become more aware of theirenergy consumption patterns and undertake activi-ties to save energy, such as changing their behav-ior or implementing energy efficient technology.From other studies, we know that the way inwhich the feedback is provided—the design ofthe feedback system—plays a role in how peopleengage with that system and whether or not energysavings are achieved (e.g., Kobus et al. 2015b;Buchanan et al. 2015). From the questionnaires,we found that the users of the smartphone/tabletapplications reported higher awareness and moreenergy-saving activities than the reference group.However, we did not find an actual effect in termsof measured gas and electricity consumptionlevels.

How can these differences in results be ex-plained? First, the results are based on differentsamples that may differ in their energy behaviors.Yet considering that no energy savings were re-vealed by an additional analysis on the more re-stricted sample of the households who participatedin both the energy measurement and questionnairestudies, this explanation does not seem likely.Another more likely explanation would be thatparticipants may have given socially desirable an-swers in the questionnaire study, meaning thatparticipants provided an over-optimistic picture oftheir energy behavior. Finally, the interview resultsindicate that the applications tended to be used tomonitor consumption levels and thereby gain moreawareness, rather than to achieve lower consump-tion levels.

Given that the designs of the apps in this studyare similar to many of the apps available in theNetherlands, it will be important in the future tolook into ways to improve their designs, so thatthey not only facilitate energy monitoring but alsoencourage energy saving. The insights from theinterviews into how people used and evaluatedthe feedback system provide some explanationsand allow for the formulation of recommendationsfor the design of an app-based feedback system.

The study results suggest that the applicationswere used to gain insight in one’s energy con-sumption levels and to monitor consumption overtime (i.e., to check whether consumption levels areas expected), rather than as a tool to save energy.The applications did not seem to offer sufficiently


concrete and actionable information to assist inenergy saving and, in the case of low motivation,to motivate undertaking energy-saving activities.These findings are in line with a study byNilsson et al. (2014) in which insufficient under-standing of the provided information and lack ofinterest in energy saving were shown to be impor-tant barriers to achieving energy savings. The ap-plication should thus provide information that isrelevant, meaningful, and actionable to users whodo not have a strong interest in understanding theirconsumption.

Hargreaves et al. (2013) described how ahouseholder may think that it is not possible tosave energy while maintaining a comfortable life-style because he or she is not aware of possibil-ities or methods for achieving energy savingswithout decreasing one’s comfort. We also foundthis in the interviews in this research. We alsofound that no energy-saving actions were taken inhouseholds where the feedback information wasused to evaluate the current situations and thehouseholders thought that the consumption levelswere acceptable, for example, because a con-sumption level was lower than that of similarhouseholds, or was within the household’s bud-get. These findings lead us to recommend that anapplication designed to achieve change in ahousehold’s energy-related behavior must guideusers towards becoming aware of how and whereto save energy in ways that match the house-hold’s needs and abilities.

The finding that the apps were used for moni-toring purposes after an initial learning phase wasalso observed in the evaluation of the smart meterroll out in the UK. The UK researchers observedthat most households that received an in-homedisplay after the installation of the smart meterwere still using it 2 years later to monitor theirconsumption. There were also energy-savings ef-fects of about 2–3% (Darby et al. 2015). In ourresearch, we found that most of the applicationusers were still using their app after more than6 months (74% in the questionnaire at T2). Soperhaps the fact that people can continue to usean app to monitor their consumption may, in time,pay off as people gradually change their behaviorand implement energy-saving measures. To gainmore insight into this potential, we suggest

looking into the effects of apps that have beenavailable in the market (paid or for free) for sev-eral years.

The feedback system was easily accessible withthe smartphone/tablet application, but this did notfacilitate frequent use of the app in the long term.We found a significant drop in use frequencybetween T1 and T2. Given that the applicationusers were self-selected to the research by apply-ing for the feedback system, we would expect abasic interest in energy consumption and thereforerecurring/frequent use of the app. If a high-usefrequency could not be sustained within this sam-ple during the research, households that are notinterested in applying for a free feedback systemmay be even less likely to use an app regularly.As we uncovered in the interviews, one reason forlow-use frequency can be the limited relevance ofthe information to the user and declining perceivedrelevance of the information after an initial learn-ing period. Additionally, an app can easily be Boutof sight, out of mind.^ People have to deliberatelylook up and open the app on their smartphone/tablet. When they are not committed to do so, ortriggered by, e.g., an icon on their home screen ornotifications, this provides a barrier to seeing andusing the feedback.

A higher use frequency was found to be relatedto more energy saving by Kobus et al. (2015b)and is suggested as well by the correlation foundin this study with respect to reported use frequen-cy and behavior change. Therefore, we recommendexploring which design elements of an app drawmore attention from users, for example, via notifi-cations with tips or reminders related to one’sactual consumption pattern, or ambient cues indi-cating via colors, or icons showing when energyconsumption is higher than expected from one’snormal consumption. Sustaining a user’s interestin the feedback information can be a challengefor energy saving via a feedback system(Buchanan et al. 2015; Darby et al. 2015; Kobuset al. 2015b). App designers should thus alsoconsider how an app can continue to be relevantas the household’s consumption patterns and inter-ests change over time. Given that apps are device-specific—unlike in-home displays—and easily up-dated, they offer a valuable opportunity in thesense that they can grow along with their users.


Another factor that may have influenced theeffectiveness of the feedback system was that theapps were only used by the Benergy manager^ ofthe household. The information about a house-hold’s energy consumption was therefore only han-dled by one person. This person would have toinitiate conversations about the household’s energyuse and to get cooperation from family membersfor investing in energy efficiency measures or tomake changes in daily routines. This barrier wasalso observed by Van Dam (2013) and Hargreaveset al. (2010). The disadvantage of an app, com-pared to an in-home display, is that it is not ashared object in the household. We have two sug-gestions for this issue: first, to make it easy forthe energy manager to share insights and ideaswith his household members, and second, to makethe use of the app more attractive and relevant forall household members. This also means that thefeedback system has to respond to the dynamics ofdaily practices in homes. In the words ofStrengers, the design has to move beyond theinterest of the rational and individual BResourceMan^ (Strengers 2014).

We found that the implementation of energy-saving measures takes time. People have to findout what energy-saving measures they want totake, and this often involves waiting for the rightmoment. A necessary replacement of appliancesand the making of home improvement plans areoften moments when energy efficiency consider-ations are more easily taken into account andexecuted (Verplanken and Wood 2006; Stieß andDunkelberg 2013). For the effects of the interven-tion in this study, this means that the feedbackmay contribute to energy saving that occurs afterthe intervention. For the design of a feedbacksystem, we recommend that it supports thedecision-making process for the implementationof energy-saving measures by complementing en-ergy consumption information with more contextu-al information and advice, e.g., about how to bestget advice tailored to your home from local ex-perts, about cost-benefit trade-offs, and aboutavailable subsidies.

Finally, it is important to reflect on what theenergy-saving potential actually was during thestudy. The energy-saving actions of the householdsmay not have had a noticeable impact on the

overall consumption levels. Several householdssuggested that their house and behavior were al-ready quite energy efficient, which leads to thequestion of how much more energy could havebeen saved with the measures that were primarilymentioned: changing light bulbs and purchasingefficient appliances. For interventions like the onein this study, it is recommendable to estimate thepotential savings beforehand. Knowing this, youcould also provide your users with more relevantand actionable information tailored to their ownsituations.

Limitations of the study

We structured our research to obtain a represen-tative sample of Dutch households. The demo-graphics of the sample do however differ tosome extent from those of the general Dutchpopulation and those of other countries. In com-parison to the Dutch population4, the sample forthis research is older and more highly educated.More of the houses are privately owned thanrented. The homes are relatively new (laterbuilt), and they also have larger floor surfaces,more occupants, and higher energy consumptionon average. Complementary research with sam-ples of different compositions could provide in-sight herein.

This research was executed in a natural settingwithin the regular process of smart meter instal-lations. The recruitment of participants for ques-tionnaires and meter readings had to be doneseparately from the meter installation and theparticipants’ application for the feedback system,in order to avoid influencing the network opera-tor’s usual way of approaching households forsmart meter installations. As a result, a self-selection bias was introduced for the applicationuser group and we were not able to work withone large sample for which both meter readingsand questionnaire results were available. Ourresearch approach did however provide a uniqueopportunity to study the effects of a feedbacksystem in a normal situation. By setting up the

4 Based on data from the Central Bureau of Statistics ofThe Netherlands. Accessed online via statline.cbs.nl in 30March 2016.


http://statline.cbs.nl

three studies, we were able to gain complemen-tary insights into the effects of the apps onhousehold energy consumption, both quantita-tively and qualitatively.

In this research, we chose to let one respon-dent represent the household, yet one respondentcannot fully represent a multi-person household.This study collected and examined the viewpointsof household energy managers. For a more com-plete picture of the role of a feedback system,future research would have to address the com-plex dynamics of households and the (energy-consuming) products and services they use. Thisis particularly relevant for finding out how prod-ucts and services can facilitate energy efficiencywithin a household’s daily practices and as relatedto its wider social context, as described, e.g., byShove (2010) , Gram-Hanssen (2010) , andSchwartz et al. (2015).

Conclusion

This research contributes to the existing literatureabout feedback systems with insights into the useand effectiveness of a smartphone/tablet app. Theapplication users in the sample with measurementof energy consumption levels did not show a de-crease in their energy consumption during the re-search period, compared to the reference group.Yet in the sample who responded to question-naires, application users reported increased aware-ness and energy-saving activities compared to thereference group. Further insight from interviewswith application users indicated that people usedthe apps mainly to learn how their energy con-sumption levels are built up and to monitor theconsumption levels over time, rather than to de-crease one’s consumption levels. In line with otherresearch into feedback, the interview results sug-gest that an app could be more effective withinformation that is more actionable and meaningfulwith respect one’s own specific situation.Furthermore, more effectiveness can be expectedwhen a higher use frequency is stimulated andinsights are provided that relate to the goals theend-users want to achieve in their household.

Based on this research, we cannot yet concludewhether apps do or do not lead to energy saving.

Further exploration is recommended with respectto how the design of such apps can encourage awide audience to monitor their consumption andguide them in taking action to change their con-sumption levels. In light of the implementation ofsmart meters and feedback systems based on meterdata, policy makers should be aware that feedbacksystems do not necessarily lead to energy savingsbecause their effectiveness depends on their de-signs and the contexts in which they are imple-mented. Feedback systems play a valuable part bymaking energy consumption visible, but a compre-hensive approach with complementary products,services, and/or policies is important in facilitatinga household’s process of learning and decisionmaking about energy efficiency. Policy makersshould take this into account when defining goals,approaches, and guidelines for stimulating energysaving.

In view of the current energy transition, withincreasingly decentralized production of renew-able energy and the growing uptake of electrictransport and heat pumps, we suggest broadeningthe discourse on smart meter implementation forhouseholds. The insight in energy consumptionand production that smart meters can provideshould go beyond encouraging households tosolely save energy, enabling them to adjust theirelectricity consumption patterns to the demandand supply of (local and renewable) energy.

Data statement

The data that support the findings of this study areavailable on request from the corresponding author.The data are not publicly available due to them contain-ing information that could compromise research partic-ipant privacy.

Conflict of interest The authors declare the following interests:

– Daphne Geelen and Annemieke Bulters are employed byEnexis B.V., a distribution network operator implementingsmart meters according to national policy in the Netherlands.

– Delft University of Technology received financial supportfrom Enexis B.V. for Ruth Mugge and Sacha Silvester to beable to contribute to the research reported in the submittedwork.


Tab

le5

Measuresfrom

questio

nnaire

Topic

Questions

T1

or T2

Answer

optio

nsResponses

[A]:forapplicationuser

group

Total

Reference

group

Application

users

App

AApp

B[R]:forreferencegroup

RequestingP1-reader

Why

didyouapplyforthe

P1-reader?[A

]T1

1.The

P1-reader

isforfree.

––

10%

––

2.Curious

aboutm

yenergy

consum

ption/w

ould

liketosave

energy

/wanttohave

moreinsight.

64%

3.Niceto

tryanewdevice/gadget.

12%

4.Other,nam

ely…

14%

Morethan

oneansw

erpossible

%of

answ

ers

N=129

Use

ofapplication

Which

oftheapplications

doyou

usemost?[A

]T1

1.App

A–

–55%

––

2.App

B29%

3.Desktop

application

12%

4.Onlineapplication

4% N=97

T2

1.App

A–

–54%

––

2.App

B34%

3.Desktop

application

12%

4.Onlineapplication

0% N=68

Why

areyounotu

sing

orhave

you

stopped

usingtheapplication(s)?[A

]

T2

…(openansw

er)

––

––

–

Whatisthemainreason

thatyouare

using

thisapplication

most?[A

]

T1

1.Design

––

17%

12%

33%

2.Fu

nctio

nality

18%

14%

27%

3.Fo

rfree

3%2%

3%

4.Independentp

rovider

0%0%

0%

5.Knownprovider

3%0%

9%

6.Firsto

neIinstalled

23%

27%

3%

7.The

only

oneavailable/thatI

found

26%

31%

15%

8.Other,nam

ely…

11%

15%

9%

Morethan

oneansw

erpossible

%of

answ

ers

(N=111)

%of answ

ers

(N=59)

%of answ

ers

(N=33)

App

endix


Tab

le5

(contin

ued)

Topic

Questions

T1

or T2

Answer

optio

nsResponses


group

Total

Reference

group

Application

users

App

AApp


Use

frequency

How

oftendo

youlook

atthe

application?

[A]

T1

1.Never

––

1%0%

0%

2.Lessthan

once

amonth

5%2%

4%

3.About

once

amonth

4%4%

7%

4.About

once

everytwoweeks

31%

36%

25%

5.About

once

aweek

27%

30%

25%

6.Afewtim

esaweek(not

everyday)

14%

11%

11%

7.Onceaday

18%

17%

29%

8.Severaltim

esaday

N=97

N=53

N=28

T2

1.Never

––

9%3%

13%

2.Lessthan

once

amonth

22%

32%

9%

3.About

once

amonth

9%11%

9%

4.About

once

everytwoweeks

22%

11%

26%

5.About

once

aweek

24%

24%

30%

6.Afewtim

esaweek(not

everyday)

9%14%

4%

7.Onceaday

6%5%

9%

8.Severaltim

esaday

N=68

N=37

N=23

Insightinenergy

consum

ption

Towhatextentd

oyouthinkthe

smartm

eter

(applicationusers:in

combinatio

nwith

the

P1-reader)provides

youbetter

insightin

your

energy

consum

ption?

[A]

[R]

T1

1.Not

20%

31%

6%6%

4%

2.Alittle

41%

44%

37%

35%

43%

3.Alot

38%

24%

54%

58%

50%

4.Quitealot

2%2%

2%2%

4%

N=204

N=110

N=94

N=52

N=28

T2

1.Not

24%

32%

14%

12%

14%

2.Alittle

41%

38%

46%

40%

46%

3.Alot

32%

26%

39%

42%

39%

4.Quitealot

3%3%

3%6%

0%

N=213

N=117

N=96

N=52

N=28

Helpwith

energy

saving

Towhatextentd

oestheapplication

help

youto

save

energy?[A

]T1

1.Not

atall

––

2%2%

0%

2.Verylittle

16%

17%

14%


Tab

le5

(contin

ued)

Topic

Questions

T1

or T2

Answer

optio

nsResponses


group

Total

Reference

group

Application

users

App

AApp


3.Alittle

37%

36%

36%

4.Alot

31%

30%

39%

5.Quitealot

13%

15%

11%

N=97

N=53

N=28

T2

1.Not

atall

––

9%11%

5%

2.Verylittle

26%

26%

29%

3.Alittle

44%

37%

52%

4.Alot

21%

26%

14%

5.Quitealot

0%0%

0%

N=68

N=38

N=21

Towhatextentd

oesthebi-m

onthly

overview

help

youto

save

energy?[A

][R]

T2

1.Not

atall

21%

28%

12%

15%

6%

2.Verylittle

25%

21%

30%

35%

25%

3.Alittle

36%

29%

47%

35%

63%

4.Alot

17%

20%

12%

15%

6%

5.Quitealot

1%1%

0%0%

0%

N=149

N=89

N=60

N=34

N=16

Behaviour

change

You

may

have

takenmeasuresto

save

energy

inyourhomeafterthe

installatio

nof

the

smartm

eter(applicationusers:in

combinatio

nwith

theP1-reader).

Towhatextenth

aveyouchanged

your

daily

behaviour?

[A][R]

T1

1.Not

atall

59%

68%

48%

53%

36%

2.Verylittle

20%

17%

23%

15%

32%

3.Alittle

13%

9%20%

23%

25%

4.Alot

7%6%

9%9%

7%

5.Quitealot

0%1%

0%0%

0%

N=224

N=127

N=97

N=53

N=28

T2

1.Not

atall

43%

54%

29%

28%

21%

2.Verylittle

21%

17%

27%

30%

29%

3.Alittle

24%

17%

32%

28%

36%

4.Alot

11%

10%

11%

11%

14%

5.Quitealot

1%1%

1%2%

0%

N=224

N=127

N=97

N=53

N=28

Haveyoureplaced

appliances

ormadechanges

T1

1.No

91%

92%

89%

87%

89%

2.Yes,nam

ely…

9%8%

11%

13%

11%


Tab

le5

(contin

ued)

Topic

Questions

T1

or T2

Answer

optio

nsResponses


group

Total

Reference

group

Application

users

App

AApp


toyourhomebasedon

theenergy

consum

ptioninform

ation?

[A]

[R]

N=224

N=127

N=97

N=53

N=28

T2

1.No

79%

88%

68%

66%

64%

2.Yes,nam

ely…

21%

12%

32%

34%

36%

N=224

N=127

N=97

N=53

N=28

Towhatextentd

oyouintend

toinvestin

your

hometo

lower

your

energy

consum

ption

levels?[A

][R]

T1

1.Not

atall

30%

39%

19%

13%

21%

2.Verylittle

34%

35%

33%

32%

36%

3.Alittle

21%

16%

29%

34%

29%

4.Alot

12%

8%18%

19%

11%

5.Quitealot

3%3%

2%2%

4%

N=224

N=127

N=97

N=53

N=28

T2

1.Not

atall

47%

56%

36%

32%

43%

2.Verylittle

28%

27%

30%

30%

39%

3.Alittle

18%

14%

24%

26%

11%

4.Alot

5%2%

9%11%

4%

5.Quitealot

1%1%

1%0%

4%

N=224

N=127

N=97

N=53

N=28

(ifintending

toinvest)W

hatkindof

investments

doyouwanttomake?

[A][R]

…(answerswerecoded)

––

––

–


OpenAccessThis article is distributed under the terms of theCreative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestrict-ed use, distribution, and reproduction in any medium, providedyou give appropriate credit to the original author(s) and the source,provide a link to the Creative Commons license, and indicate ifchanges were made.

Publisher’s note Springer Nature remains neutral with regard tojurisdictional claims in published maps and institutional affilia-tions.

References

Abrahamse, W., Steg, L., Vlek, C., & Rothengatter, T. (2005). Areview of intervention studies aimed at household energyconservation. Journal of Environmental Psychology, 25,273–291.

Anderson, W., & White, V. (2009). Exploring consumer prefer-ences for home energy display functionality. Report to theE n e r g y S a v i n g Tr u s t , 1 2 3 . www. c s e . o r g .uk/app/webroot/downloads/file/consumer_preferences_for_home_energy_display.pdf. Accessed 28 Jan 2019.

Bonino, D., Corno, F., & De Russis, L. (2012). Home energyconsumption feedback: a user survey. Energy andBuildings, 47, 383–393.

Buchanan, K., Russo, R., & Anderson, B. (2015). The question ofenergy reduction: the problem(s) with feedback. EnergyPolicy, 77, 89–96.

Darby, S. (2006). The effectiveness of feedback on energy con-sumption. A review for DEFRA of the literature on metering,billing and direct displays. Environmental Change Institute,University of Oxford. A Review for DEFRA of the Literatureon Metering Billing and direct Displays, 22nd. www.volksmeter.de/Abhandlungen/Darby2006.pdf. Accessed 28Jan 2019.

Darby, S., Liddell, C., Hills, D., &Drabble, D. (2015). Smart meteringearly learning project: synthesis report. United Kingdom:Department of Energy and Climate Change (DECC).

Ehrhardt-Martinez, K., Donnelly, K. A., & Laitner, J. A.(2010). Advanced metering initiatives and residentialfeedback programs: a meta-review for householdelectricity-saving opportunities. Washington: Reportfor ACEEE.

European Commission (2017). Energy Efficiency Directive.h t t p s : / / e c . e u r opa . eu / ene rgy / en / t op i c s / ene rgy -efficiency/energy-efficiency-directive. Accessed 2 Jan 2017.

European Union (2012). Directive 2012/27/EU of the EuropeanParliament and of the Council of 25 October 2012 on energyefficiency, amending Directives 2009/125/EC and 2010/30/EU and repealing Directives 2004/8/EC and 2006/32.Official Journal, L, 315, 1–56.

European Union. (2016). EU energy in figures – Statistical pock-etbook 2016. Luxembourg: Publications Office of theEuropean Union ISBN 978-92-79-58247-9.

Fischer, C. (2008). Feedback on household electricity con-sumption: a tool for saving energy? Energy Efficiency,1, 79–104.

Geelen, D., Reinders, A., & Keyson, D. (2013). Empowering theend-user in smart grids: recommendations for the design ofproducts and services. Energy Policy, 61, 151–161.

Giordano, V., Gangale, F., Fulli, G., & Sánchez Jiménez, M.(2011). Smart grid projects in Europe: lessons learned andcurrent developments. Petten: Joint Research Centre of theEuropean Comission.

Gram-Hanssen, K. (2010). Standby consumption in householdsanalyzed with a practice theory approach. Journal ofIndustrial Ecology, 14(1), 150–165.

Guest, G., Bunce, A., & Johnson, L. (2006). Howmany interviewsare enough? An experiment with data saturation and variabil-ity. Field Methods, 18(1), 59–82.

Hagaman, A., & Wutich, A. (2017). How many interviews areenough to identify Metathemes in multisited and cross-cultural research? Another perspective on Guest, Bunce,and Johnson’s (2006) landmark study. Field Methods,29(1), 23–41.

Hargreaves, T., Nye, M., & Burgess, J. (2010). Making energyvisible: a qualitative field study of how householders interactwith feedback from smart energy monitors. Energy Policy,38(10), 6111–6119.

Hargreaves, T., Nye, M., & Burgess, J. (2013). Keeping energyvisible? Exploring how householders interact with feedbackfrom smart energy monitors in the longer term. EnergyPolicy, 52, 126–134.

Karjalainen, S. (2011). Consumer preferences for feedback onhousehold electricity consumption. Energy and Buildings,43(2), 458–467.

Karlin, B., Zinger, J. F., & Ford, R. (2015). The effects of feedbackon energy conservation: a meta-analysis. PsychologicalBulletin, 141(6), 1205.

Kobus, C., Klaassen, E., Mugge, R., & Schoormans, J. (2015a). Areal-life assessment on the effect of smart appliances forshifting households’ electricity demand. Applied Energy,147(June), 335–343.

Kobus, C., Mugge, R., & Schoormans, J. (2015b). Long-term influence of the design of energy managementsystems on lowering household energy consumption.International Journal of Sustainable Engineering,8(3), 173–185.

McKerracher, C., & Torriti, J. (2013). Energy consumption feed-back in perspective: integrating Australian data to meta-analyses on in-home displays. Energy Efficiency, 6(2), 387–405.

Nilsson, A., Bergstad, C. J., Thuvander, L., Andersson,D., Andersson, K., & Meiling, P. (2014). Effects ofcontinuous feedback on households’ electricity con-sumption: potentials and barriers. Applied Energy,122, 17–23.

Ofgem & DECC (2011). Smart Metering ImplementationProgramme - Response to Prospectus Consultation,Overview Document. Available from: https://www.ofgem.gov.uk/sites/default/files/docs/2011/03/sm-prospectus-reponse-overview_0.pdf. Accessed 28 Jan 2019.

Schwartz, T., Stevens, G., Jakobi, T., Denef, S., Ramirez, L.,Wulf,V., & Randall, D. (2015). What people do with consumptionfeedback: a long-term living lab study of a home energymanagement system. Interacting with Computers, 27(6),551–576.


https://www.cse.org.uk/app/webroot/downloads/file/consumer_preferences_for_home_energy_display.pdf



https://www.volksmeter.de/Abhandlungen/Darby2006.pdf


https://ec.europa.eu/energy/en/topics/energy-efficiency/energy-efficiency-directive

https://ec.europa.eu/energy/en/topics/energy-efficiency/energy-efficiency-directive

https://www.ofgem.gov.uk/sites/default/files/docs/2011/03/sm-prospectus-reponse-overview_0.pdf



Shove, E. (2010). Beyond the ABC: climate change policy andtheories of social change. Environment and Planning A,42(6), 1273–1285.

Stankovic, L., Stankovic, V., Liao, J., & Wilson, C. (2016).Measuring the energy intensity of domestic activities fromsmart meter data. Applied Energy, 183, 1565–1580.

Stieß, I., & Dunkelberg, E. (2013). Objectives, barriers and occa-sions for energy efficient refurbishment by privatehomeowners. Journal of Cleaner Production, 48, 250–259.

Strengers, Y. (2014). Smart energy in everyday life are you de-signing for resource man? Interactions, 21(4), 24–31.

Van Dam, S. (2013). Smart energy management for house-holds. Doctoral thesis. Delft: Delft University ofTechnology.

Van Dam, S. S., Bakker, C. A., & Van Hal, J. D. M. (2010). Homeenergy monitors: impact over the medium-term. BuildingResearch & Information, 38(5), 458–469.

Van Dam, S. S., Bakker, C. A., & Van Hal, J. D. M. (2012).Insights into the design, use and implementation of homeenergy management systems. Journal of Design Research,14, 10(1–2), 86–101.

Van Elburg, H. (2014).Monitor Energiebesparing Slimme Meters(Besparingsmonitor). Report Rijksdienst voor OndernemendNederland. https://www.google.nl/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=2ahUKEwjn_Lqj65DgAhUSLFAKHQm7C9IQFjAAegQICRAC&url=https%3A%2F%2Fzoek.officielebekendmakingen.nl%2Fblg-301727.pdf&usg=AOvVaw1oYvtvmsPYtgXEY_tlMng3.Accessed 28 Jan 2019.

Verplanken, B., & Wood, W. (2006). Interventions to make andbreak consumer habits. Journal of Public Policy &Marketing, 25(1), 90–103.







the use of apps to promote energy saving: a study of smart … · 2019. 8. 10. · original article...

Documents