gcn evaluation guide
TRANSCRIPT
-
7/30/2019 GCN Evaluation Guide
1/50
Evaluating GovernmentCommunication ActivityStandards and Guidance
-
7/30/2019 GCN Evaluation Guide
2/50
Evaluating Government Communication Activity
2
Contents
Introduction 3
PROOF: ve guiding principles or evaluation 4
The big IDIA: the our-stage evaluation process 5
Stage 1: Identiy 6
Stage 2: Develop 8
Stage 3: Implement 17
Stage 4: Analyse and report 23
Conclusions 27
Appendix: Recommended metrics 28
-
7/30/2019 GCN Evaluation Guide
3/50
3
Evaluating Government Communication Activity
IntroductionAs government communicators, were all aware o the need to make every piece o work
that we produce as eective and ecient as possible. To do this, we need to understand
whats already working well and where theres room or improvement. This in turn requires
us to evaluate our work and apply the learning rom this evaluation to uture activity.
Were also increasingly required to demonstrate how were applying evaluation in our
day-to-day jobs, through the plans and reports that we submit to the Eciency and
Reorm Group (ERG) or activities with a spend o 100,000 or more and the inormation
that we supply to eed into the annual communication plan. The revised GovernmentCommunication Network (GCN) Core Skills or Government Communicators will set out
clear evaluation standards that we should all ollow, based on our grade and the discipline
that we work in.
To help us evaluate our activity eectively, in line with expected standards, we need clarity
on what good evaluation practice looks like. This guide sets out an approach to evaluation
that should be ollowed as a minimum or all government communication activity,
regardless o size, discipline or budget. This approach is pragmatic and ocuses on helping
you to produce the best possible evaluation given the scope o your activity and the time,
resource and budget that you have available or evaluation. By ollowing the guide, you
can be condent that you will produce an evaluation that meets the required standards or
your activity and or your role.
I youre new to evaluation, use the guide to help you get started, recognising that it will
take time to build up your approach and gather whats working. Remember, a partial
evaluation is almost always better than no evaluation at all. I youre already evaluating
your activity eectively, ensure that youre ollowing the standards required o you, making
modications as necessary. Think about how you can share what youve learned with
others to help them evaluate more eectively.
By evaluating the activity that we carry out, we will be able to improve the eectiveness
and eciency o our work over time in the uture. We will also be able to demonstrate the
contribution that well-planned and executed communication activity makes to governmentoverall and hence justiy urther investment in our work.
https://gcn.civilservice.gov.uk/gcn-core-skills/https://gcn.civilservice.gov.uk/gcn-core-skills/ -
7/30/2019 GCN Evaluation Guide
4/50
4
Evaluating Government Communication Activity
P Pragmatic best availablewithin budget, not best ever
A pre-planned, but partial, evaluation is better than no evaluation at all. Be transparent: acknowledge the gaps in your
evaluation and the implications o these gaps. Even i you are not able to ully quantiy the eect o your communication
activity, you will still be able to draw valuable learning rom the evidence that you have obtained.
R Realistic prove what you can,acknowledge what you cant
Dont worry i you can only collect a small amount o data in the short term. By establishing a robust evaluationramework that is linked to a clear set o communication objectives, you will be able to interpret and analyse whatever
inormation you gather. Over time, you will be able to build on this knowledge, increasing the amount o data that you
collect rom each subsequent activity.
O
Objective approach yourevaluation with an open mind
Be honest and constructive about what was achieved, so that we can all learn or the uture. Learn rom your successes
and rom things that didnt work as well as youd hoped. Use this to rene your uture strategy.
O Open record and shareas much as possible
Share your learning as widely as possible so that colleagues can also benet rom your experience. Work with GCN to
develop a detailed case study.
F Fully integrated integrate evaluationinto activity planning and delivery
Plan ahead. Start thinking about how to evaluate your activity as soon as possible, ideally well beore it begins. This will
help you to put the right mechanisms in place or subsequent measurement and data collection. Retrospective evaluation
is oten less eective because the right data may not have been collected or objectives may not be measurable.
PROOF: fve guidingprinciples or evaluation
Whatever the size or scope o your communication activity,ollowing these guiding principles will help to ensure thatyour evaluation is as eective as possible.
-
7/30/2019 GCN Evaluation Guide
5/50
Evaluating Government Communication Activity
5
You can evaluate all kinds o communication activity,including press and media management, marketing andinternal communication. Follow the our-stage process set out
in Figure 1 below and you will carry out an eective evaluationin line with the ve guiding principles or evaluation.
Figure 1: The our-stage evaluation process
The big IDIA:the our-stage evaluation process
1
2
3
4
Identiythe scope o your project
Developyour evaluation plan
Implement source data to measure perormance
Analyse and reportperormance against the plan
-
7/30/2019 GCN Evaluation Guide
6/50
Evaluating Government Communication Activity
6 Stage 1: Identiy
Stage 1: Identify thescope o your project
Checklist
Task: to dene what you need to evaluate by asking:
What activity am I evaluating?
What do I already know?
What is my evaluation expected to achieve?
Output: summary o your proposed evaluation approach
What activity are you evaluating?
Begin by establishing exactly what you are going to evaluate. What activities do you need
to include? Are these activities part o a wider communication strategy?
Identiy the time period over which you will evaluate the activity. For a one-o piece
o activity with clear start and end dates, this is generally straightorward. For ongoing
activity, you need to identiy and agree appropriate time periods or evaluation, i.e. how
oten you will assess perormance against objectives.
Examples o types o activity
Press For ongoing reputation management work, its oten most
eective to track the eect o your work over time. Identiy the key
messages that you want to track and monitor coverage on a regular
basis, providing monthly or quarterly updates rather than reporting
on individual activities.
Marketing I the activity that you are evaluating runs across a range o
channels (which might include, or example, paid-or activity,
partnerships, leafets, a website or social media), check that youve
included all o them in your plan.
Internal
communication
I you are evaluating the eect o a change management
programme, ensure that you include all the elements o the
programme in your evaluation plan (e.g. briengs to senior sta,
communication via email and the intranet, events, training etc).
-
7/30/2019 GCN Evaluation Guide
7/50
Evaluating Government Communication Activity
Stage 1: Identiy 7
What do you already know?
Review similar previous activity that your team or others have run to see what you can
learn. This will give you a benchmark to measure perormance against and may help to
rene your objectives and your approach to the activity that you are about to run.
$ Useful tip gathering evidence or ERG exemption requests
I your activity is subject to ERG approval, use your previous evaluation results
as evidence or why you believe it will meet its objectives in sections 3 and
10 o the exemption request orm.
What is your evaluation expected to achieve?
Ask the ollowing questions to help identiy what is expected rom your evaluation:
Whatarethekeyquestionsthatyourevaluationreportneedstoanswer?
Whatlevelofdetailisexpectedfromyourreportandwhatformatdoesitneedtobein?
Isanybudgetavailableforyourevaluation?
Whowilldothework?Thismightincludeyouandothersonyourteam,research,
analysis and evaluation specialists rom within your hub or the Shared Communications
Service, an external agency or a combination o all o these.
$ Useful tip ERG evaluation standards
I your activity is subject to ERG approval, there is a standard ormat that you
must use or your evaluation report. A report template can be ound here.
Ensure that your evaluation plans are designed with this in mind
Summarising your approach to evaluation
Beore beginning work, set out your proposed approach to evaluation, basing this on the
inormation that you have gathered so ar. Depending on the scope o the activity that
youre evaluating and the person or whom you are carrying it out, this may be a detailed
proposal to stakeholders in your department, an email to your manager or a note to
yoursel. Whatever ormat you use, you can nd a useul template here that sets out the
questions you should consider.
$ Useful tip managing expectations
Sometimes, there may be a gap between the time, resource and budget
that you have available and others expectations o what your evaluation can
achieve. Discuss whether its better to increase available resource, budget or
time or to lower expectations. Decide this early in the process.
https://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/ERG-Evaluation-report-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-approach-template.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-approach-template.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/ERG-Evaluation-report-template-.doc -
7/30/2019 GCN Evaluation Guide
8/50
Evaluating Government Communication Activity
8 Stage 2: Develop
Stage 2: Develop yourevaluation plan
Checklist
Task: to dene how youll measure
success by:
Setting out activity objectives
Dening target audiences
Mapping out how your activitywill work
Setting perormance metrics
Agreeing metrics and targets
Output: drat evaluation plan
Setting out activity objectives
Clear and measurable communication objectives are the cornerstone o any evaluation
plan. They set out what your activity aims to achieve and the overall goals against which
you should judge success.
Your objectives should already be identied in the communication strategy or the
activity you are evaluating. It may be useul to summarise them as shown in Figure 2,
clearly demonstrating how each sub-objective links to the overall communication and
departmental objective.
Startbyidentifyingtheoveralldepartmental objective and the issue it is designed
to address. Your activity will be part o a set o interventions which link back to this
objective.
Next,identifythecommunication objective. This is the overall role that
communication is expected to play in achieving the policy objective.
Differentchannelsoractivitiesmayhavedistinctrolestoplayinachievingtheoverall communication objective. Each o these should be set out as a separate
communication sub-objective.
You may only be evaluating the eectiveness o one communication sub-objective, but
it is important to understand how this is expected to contribute towards the overall
communication objective.
Ensure that all communication objectives are clear and set out what each activity was put
in place to achieve, together with a measure o success. Always consider whether or not
you will be able to prove a communication objective has been met. I not, it will need to
be revised.
You may nd it useul to map out your objectives as shown in Figure 2.
-
7/30/2019 GCN Evaluation Guide
9/50
Evaluating Government Communication Activity
Stage 2: Develop 9
Figure 2: Hierarchy o objectives
Sample objectives might include:
Press Departmental objective: To ensure that compliance with a new tax
regulation is above 80%.
Communication objective: To ensure that the majority o the general
public understands the reasons why complying with the regulation
benets the economy.
Press-specic sub-objective: To ensure that the public is given a air
and balanced view o the policy, via the media.
Marketing Departmental objective: To get 10,000 more people working as
community service volunteers in your area.
Communication objective: To get 40,000 people in the area to
register as potential volunteers on your community website.
Sub-objective 1: To increase the proportion o the public who
recognise the value o volunteering rom 20% to 40%.
Sub-objective 2: To get 80,000 people to visit the website and nd
out more about how to volunteer.
Sub-objective 3: To secure 40,000 incremental registrations.
Internal
communication
Departmental objective: To ensure that unauthorised sta absences
all by 50%.
Communication objective: To ensure that all sta are able to ollow
the correct processes or reporting absences rom work.
Sub-objective 1: To ensure that all sta recognise that there is a policyor reporting absences rom work and that they must ollow this.
Sub-objective 2: To ensure that all sta understand how to access the
guidance on how to report absences.
Departmentalobjective
Put in place to address
a specic issue
Includes: policy development,
policy delivery, reputationmanagement
Communicationobjective
Role that communicationwill contribute to
achieving departmental
objective
Communicationsub-objectives
Role that individualactivities or channels
will play in meeting the
overall communicationobjective
-
7/30/2019 GCN Evaluation Guide
10/50
10 Stage 2: Develop
Evaluating Government Communication Activity
Dening target audiences
All communication activity has an end audience the people at whom it is ultimately
targeted. But an activity may also have an intermediary audience a group o people
targeted so that they will deliver the message to the end audience on your behal. Typical
intermediaries include the media, stakeholders, such as non-governmental organisations
and charities involved in delivering a policy objective, and commercial partners working
with you to deliver a piece o marketing activity.
I your activity includes an intermediary audience, you should evaluate:
howeffectivelytheintermediarywasengagedbytheactivity
howeffectivelytheintermediarycommunicatedthemessagetotheendaudience.
Audiences might include:
Press Where you are using a media engagement or PR campaign to try and
raise volunteer levels among the public overall:
Intermediary (the press): How did the media react to the activity
targeted at them? Did they eature your messages, what volume
and quality o coverage did you get or the story?
End audience (the general public): How many people volunteered asa result?
Marketing Where you are trying to engage a partner to run events promoting
volunteering on your behal:
Intermediary (partner): How eectively did you engage the partner
and how many events did they run as a result?
End audience (the general public): How many people volunteered as
a result?
Internal
communication
I you are training sta on how to communicate better with
members o the public so that customer satisaction improves:
Intermediary (sta): How eectively did you train them? Did they
put their skills into practice in their interaction with the public?
End audience (the general public): Did they notice that they received
a better service rom trained sta? Were they more satised?
$ Useful tip intermediary audiences
I your activity includes an intermediary audience, make sure you include
perormance metrics that enable you to: (1) evaluate the eect o youractivity on the intermediary; and (2) evaluate the eect o their activity on
your end audience.
-
7/30/2019 GCN Evaluation Guide
11/50
Evaluating Government Communication Activity
Stage 2: Develop 11
Mapping out how your activity will work
Spend some time thinking about how your activity is expected to achieve its objectives. I
it is successul, what messages will the target audience(s) see, what will they think or eel
and what will they do? Mapping out the steps to success will help you identiy the right
perormance metrics to evaluate your activity. Draw on any behavioural insight1 modelling
or customer journey work that has already been done.
Setting perormance metrics
Having identied the objectives and target audiences or your activity and mapped out
how you expect it to work, you need to build a set o performance metrics. These are
the measures you will use to assess the activitys perormance against its objectives and to
identiy which elements o the activity were most and least successul.
Make sure that your evaluation plan includes a range o perormance metrics rom the
ollowing ve categories:
Figure 3: The ve types o perormance metrics or evaluation
Using these ve categories as a guide will help you to pick the right perormance metrics
or your activity. The categories can be applied equally to the ull range o press, marketing
and internal communication activity.
1 For more on behavioural insight, see MINDSPACE: Infuencing behaviour through public policy (Institute or Government/Cabinet Oce www.instituteorgovernment.org.uk/our-work/better-policy-making/mindspace-behavioural-economics)
1. InputsThe activity carried out
2. OutputsHow many people had the opportunity
to see or hear your activity?
3. Out-takesWhat was its immediate eect on them?
4. Intermediate
outcomesDid they do anything as a result o your activity?
5. OutcomesDid you achieve your overall objective?
-
7/30/2019 GCN Evaluation Guide
12/50
Evaluating Government Communication Activity
12 Stage 2: Develop
Inputs
Inputs include details o the actual activity that has been undertaken, including the
channels that you used to communicate. The channels to include will have been identied
at Stage 1.
Examples o input metrics
Press Number o press releases sent out or engagement work carried out
Marketing Paid-or media plan
Website or digital space created
Number o partners contacted, types o message shared or requests
made o partners (intermediary audience)
Internal
communication
Number o sta events organised
Number o briengs or training sessions organised
Web content created and put on to the intranet
Include the costs o carrying out the activity in your input metrics. Include all externaland internal costs and time spent (including sta time). When comparing results or
more than one piece o activity, use a consistent methodology to record the costs and
time spent against each one. This will be essential or calculating return on marketing
investment.2
$ Useful tip choosing the right perormance metrics
At this stage, dont worry about whether you can get data or the
perormance metrics you choose. Pick those that you would need or an
eective evaluation. Stage 3 looks at how to secure data and deal with gaps.
2 For more inormation reer to Evaluating the nancial impact o public sector marketing communication: An Introduction to Payback, Return onMarketing Investment (ROMI) and Cost per Result (https://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/intro-to-payback-romi-and-cpr.pd)
-
7/30/2019 GCN Evaluation Guide
13/50
Evaluating Government Communication Activity
Stage 2: Develop 13
Outputs
Output metrics measure the number o people who had the opportunity to see or hear
your activity, regardless o whether they recall or recognise it. Try to include:
Reach the total number o people or organisations in your target audience who were
exposed to your activity.
Frequency the number o times they saw or heard the activity.
Examples o output metrics
Press Number o pieces o coverage achieved
Frequency o exposure to coverage by end audience
Marketing Proportion o the target audience reached by media activity
Number o impressions (one page-view) on the web page
Number o stakeholders you contacted and number o contacts
made
Internal
communication
Number o sta attending events and training sessions
Number o impressions on the intranet
$ Useful tip activity mapping
Use the activity map that you created earlier to help you identiy the right
out-take, intermediate outcome and outcome perormance metrics or
your activity.
Out-takes
Out-take metrics look at the impact that the activity had on your target audiences awareness,understanding and attitude. Think about what you wanted people to recall, think or eel about
your activity and include perormance metrics that allow you to measure this.
$ Useful tip other metrics
Perormance metrics or out-takes, intermediate outcomes and nal
outcomes cannot be standardised in the same way as those or inputs and
outputs. They will need to be tailored to refect how you expect your activity
to work and what it is trying to achieve.
-
7/30/2019 GCN Evaluation Guide
14/50
Evaluating Government Communication Activity
14 Stage 2: Develop
Examples o out-take metrics
Recall How many people are aware o your activity or the message(s) that
it is promoting?
Think How many people understand the key messages that your activity is
trying to get across?
Feel What eect has your activity had on peoples attitudes? Do they
intend to behave dierently as a result o your communication?
Intermediate outcomes
Intermediate outcome metrics capture any action taken by the target audience as a result
o your activity which may lead to the eventual end behaviour. Think about including
perormance metrics that allow you to measure the ollowing:
Examples o intermediate outcome metrics
Talk How many people discussed the activity or its message with peers,
riends and amilies? How many partners or stakeholders discussed
it with their work colleagues?
Direct response How many people responded to or otherwise interacted directlywith you as a result o the activity? This could include visiting a
website, attending training, ringing a phone line or having a
ace-to-ace discussion.
Indirect response How many people responded to or otherwise interacted with third
parties as a result o your activity? This could include people interacting
with stakeholders or partners or other local and national services.
Other actions How many people took (or claim to have taken) any other action as
a result o your activity? For example, in a stop-smoking campaign
this might include people buying books, patches or other similar
products.
Reer back to your activity map and make sure that you include measures to cover the ull
range o actions that people could have taken.
-
7/30/2019 GCN Evaluation Guide
15/50
Stage 2: Develop 15
Evaluating Government Communication Activity
Outcomes
Outcome metrics look at the eect that an activity has had on the overall communication
and policy objectives that it was put in place to address. Include metrics that enable you
to measure whether your communication objective was met and the eect your activity
had on the wider policy objective. This could be related to changing behaviour, adopting
a service, increasing positive reputation, increasing understanding and awareness or
increasing participation.
Y The appendix to this document provides a list o recommended metrics oreach component. You can use this to help identiy the appropriate metricsor your evaluation plan. This will enable you to compare your activity with
other cross-hub work and will also ensure that your plan meets the standards
required by ERG where applicable.
Agreeing KPIs and targets
You may decide to set a small number o key perormance indicators (KPIs) based on
the perormance metrics that you have chosen or KPIs may already have been set by
your team, department or hub. KPIs are measures o success that can help you track
how well an activity is progressing towards its end objective or contributing to a broader
communication strategy.
KPIs are particularly useul where your activity wont achieve its end objective or some
time. They will enable you to track how eectively the activity is progressing towards
this end objective and may provide timely inormation to make changes to the existing
communication plan, i necessary. KPIs are easier to set when the activity has run a
number o times beore, as you will have a better eel or which metrics most accurately
predict how it will ultimately perorm.
KPIs may be single perormance metrics or combine several metrics.
Youmaychoosetobringtogetherresponsesbyphone,weborface-to-facetogivea
total number of responses.
Wherepeoplesoverallsatisfactionisdrivenbyfourorvedifferentfactors,youcould
bring these together in one composite satisfaction measure.
How to set KPIs
YoudontneedKPIstoevaluateeffectivelyonlysetthemiftheyareusefulin
helping you understand whether an activity is progressing as expected or contributing
eectively to overall goals.
EnsurethatyourKPIsaremeasurable.
SelectnomorethanveKPIsforyourendaudience(andthesameforanyintermediary
audiences). SettargetsforeachKPIandspecifythetime-frameinwhichyouexpecttoachieve
them.
-
7/30/2019 GCN Evaluation Guide
16/50
Evaluating Government Communication Activity
16 Stage 2: Develop
Maketargetsasmeaningfulandasrealisticaspossible,drawingonpreviousresults
where applicable. The less historical data you have available, the broader your targets
should be.
Asfaraspossible,benchmarktargetsagainstotheractivitycarriedoutbyyouand
others within your team, department or hub. This will give you a broader context or
what success looks like.
Creating the evaluation plan
Your evaluation plan should bring together:
theobjectivesandtargetaudiencesthatyouwillevaluateperformanceagainst
theperformancemetrics(andKPIsandtargetsifyoureusingthem)thatyouwillusein
your evaluation.
Your team or department may already have a standard template or evaluation plans.
I not, you may nd these templates or activities with and without an intermediary
audience useul.
$ Useful tip multi-channel activity
For more complex activities, include a separate set o perormance metrics
or each activity/channel and audience.
https://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/Evaluation-plan-template-2.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/Evaluation-plan-template-2.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.doc -
7/30/2019 GCN Evaluation Guide
17/50
Stage 3: Implement source datato measure perormance
Checklist
Task: to identiy and gather evaluation
data by:
Identiying available data andevidence
Creating proxies and assumptions
Using monitoring, market researchand eedback
Reviewing any remaining gaps
Agreeing who will collect data
Completing the evaluation planOutput: completed evaluation plan
At this stage, you need to source data and evidence or the perormance metrics you
identied in Stage 2. Budget, time and resource may restrict the amount o data you are
able to gather but this should not stop you evaluating your activity. Its better to produce
an evaluation report with gaps in it than to produce nothing at all, provided that you are
clear about what is missing.
Identiying available data and evidence
Begin by identiying the perormance metrics or which data is immediately available. This
will come rom three main sources:
datagatheredfromyouractivity
datagatheredbystakeholdersandpartners
existingdatasources(e.g.governmentdataandwidermediaandlifestyledata).
Data gathered rom your activityGather as much data as possible directly while your communication activity is running.
Identiy all the ways in which people can respond to it and ensure that data is being
gathered or each one. Data might include web visits, telephone calls or ace-to-ace
interaction. Also, look or ways to gather additional data rom these responses or
example, asking or peoples personal details (while remaining mindul o the requirements
o the Data Protection Act 1998) or asking or permission to contact them again or
ollow-up research.
$ Useful tip plan ahead
Always try to identiy the data sources that you plan to use in your evaluation
beore the activity actually runs. This will give you time to ensure that theright data is being gathered in the right ormat while the activity is live.
Evaluating Government Communication Activity
Stage 3: Implement 17
-
7/30/2019 GCN Evaluation Guide
18/50
Evaluating Government Communication Activity
18 Stage 3: Implement
Data gathered by stakeholders and partners
Look at what data or evidence could be collected by an agency, stakeholder, partner
or colleague working with you to deliver the activity. This might include the number o
responses to an event or helpline run by a partner; any eedback received by stakeholders;
data rom surveys; competitions or promotions; and website statistics.
$ Useful tip getting data rom agencies
When youre procuring an agency to work on planning, running or
evaluating an activity or you, always ask what data they can provide to eed
into the evaluation at the procurement stage. This can then be built into their
contract. Ask or data well in advance and, where appropriate, agree to sharethe results o your evaluation.
Existing data sources
As well as resh data gathered rom current activity, you may nd it useul to look at data
sources that already exist to help you put your results in context and back them up with
supporting evidence.
For example:
Media consumption surveys such as NRS (newspapers), BARB (TV) or comScore (digital
media) provide inormation on the number o people reached by dierent media channels.
These are particularly useul or input and output measures.
Syndicated consumer lifestyle and media surveys such as TGI, TouchPoints or ACORN
can be used to build up liestyle and behavioural proles or a range o audiences. These
are particularly useul or tracking longer-term outcome measures and optimising uture
campaign planning.
Existing government demographic and research data gathered centrally or by
individual departments can be used or measuring longer-term outcomes. The Oce or
National Statistics publishes a number o surveys across all sectors o public interest.
For more inormation on how you might be able to use and access these and other
surveys, talk to colleagues in your departments or hubs research and analytics teams, the
Shared Communications Service or external media and research agencies.
Creating proxies and assumptions
I you cant get the exact data that you need or a perormance metric, look at whether
you can source a similar piece o evidence as an alternative. This is known as a proxy
measure. For example:
ifanactivityisaskingpeopletochecktheirsmokealarmsmoreregularly,agood
proxy would be the number o nine-volt batteries being sold (a type o battery almostexclusively used in smoke alarms).
-
7/30/2019 GCN Evaluation Guide
19/50
Evaluating Government Communication Activity
Stage 3: Implement 19
Is your activity similar to other activity run in the past or which youve got accurate
results? I so, you might create an assumption that this activity will perorm in the same
way. For example:
ifyouractivityencouragespeopletosignupforastop-smokingservice,youmaynot
have the budget to carry out research to see how many people have given up smoking
as a result o using the service. However, all things being equal, you could assume that
cessation rates were the same as in previous research studies.
Using monitoring, market research and eedback
I gaps still remain in your evaluation plan once you have exhausted all the available data
sources, consider lling them using bespoke monitoring, market research and eedback.
Monitoring
You can use monitoring to track how many people your communication is reaching. The
main monitoring techniques and tools include:
PR and media monitoring: I your activity is designed to generate media coverage,
you could use an agency to track how many people it reached, how many times the
key messages were mentioned and how avourable the coverage was. I you dont have
the budget or this, consider whether you could monitor coverage yoursel, either by
subscribing to a PR-monitoring service such as Gorkana or by looking at whats being saidacross a representative selection o media channels.
Web monitoring: The Government Digital Service (GDS) measures a range o standard
digital metrics or gov.uk and other government and partner sites. Topline data is available
in the regular reports provided by GDS and more detailed analysis will be available on
request. I you are responsible or monitoring perormance o a stand-alone website, there
is a range o ree and paid-or tools that you can use to monitor perormance, including
Google Analytics.
Social media monitoring: GDS will also provide guidance on social media monitoring.
There are various tools that you can use to monitor perormance yoursel:
WhereyouractivityishostedonasocialmediasitesuchasFacebook,LinkedInor
YouTube, you can set up standard user reports to monitor interactions with your
content. Look both at how many people interact with you and at the quality o these
interactions.
IfyoushareinformationviaTwitter,monitorhowmanypeoplefollowyouandhowmany
retweet your messages. Tweetdeck enables you to analyse interactions more eectively.
Buzz-monitoringtoolsenableyoutoseewhetherpeoplearecommentingonlineabout
your message, and whether the comments are positive and rom credible sources.
Eective buzz-monitoring relies on you dening the terms that you want to monitor in
advance; the more specic you can be, the more accurate the results. There is a wide
range o ree and paid-or buzz-monitoring tools. Google Alerts is one example o aree tool.
Thereisarangeoffreeandpaid-fortoolsthatcanbeusedtomonitorwhatterms
people are searching or online. Google Trends provides a useul ree snapshot.
http://www.tweetdeck.com/http://www.tweetdeck.com/ -
7/30/2019 GCN Evaluation Guide
20/50
20 Stage 3: Implement
Evaluating Government Communication Activity
Market research
I you have gaps in the data or out-take, intermediate outcome and outcome metrics,
then paid-or market research is generally the most eective way o lling them, provided
that budget is available. Market research or evaluation doesnt necessarily have to involve
large-scale ace-to-ace quantitative surveys. Lower-cost research methodologies can be
a useul source o insight. These include:
Omnibus studies: An omnibus study is a quantitative survey o a representative sample
o an audience (usually, the general public). The questionnaire is made up o groups o
questions placed by dierent clients, which means the overall costs are shared, making
it a relatively cheap option. Omnibus surveys may not be appropriate i your activity is
localised, your audience is niche, or i you want to ask many and/or in-depth questions
about a particular topic.
Commissioned online surveys: Standard online surveys recruit respondents rom large
panels o people who have agreed to take part in research. They can be very cost-eective
i your target audience is digitally engaged. However, check the quality o the panel and
how it is managed in advance. Online panels are not always suitable or tracking long-term
activities, as you may not wish to survey the same panel members repeatedly.
Qualitative research: Qualitative research, including discussion groups or interviews,
can be a useul alternative to quantitative research in some circumstances. It may be
appropriate or small-scale activities; when audiences are hard to reach; or when activity isonly running or a short period with no requirement to track its impact over time.
When commissioning paid-or research, you may be able to lower your costs by reducing
the size or specication o your sample, shortening the length o your questionnaire
or simpliying your reporting requirements. Research and evaluation specialists in your
marketing hub or the Shared Communications Service (SCS) can provide advice and help
with commissioning paid-or research or conducting your own online research.
Feedback
Feedback is inormal comment and opinion that you gather yoursel. It can be a valuable
alternative to robust research when there are limitations in time, resource and budget. This
may take the orm o a ew inormal telephone or ace-to-ace interviews with the primary
target audience, those involved in delivering the activity or direct engagement with the
audience; alternatively, eedback can be obtained through an online survey that you run
yoursel, or via a blog, email or text.
I possible, seek input rom colleagues in research or insight roles within your department
or SCS beore gathering eedback yoursel. They will be able to give you advice on
questionnaire design, data protection and propriety issues. They will also be able to advise
you on how to use online survey tools such as SurveyMonkey when seeking eedback.
-
7/30/2019 GCN Evaluation Guide
21/50
Stage 3: Implement 21
Evaluating Government Communication Activity
Examples o eedback include:
Press Speaking to a ew journalists
Monitoring the type o enquiries that you get when you run a story
Marketing Counting the number o people attending an event
Inormally asking stakeholders or partners or their observations on
how an activity went
Running a survey on your website or social media space
Adding a question about your service to a call-centre script
Internal
communication
Inormal interviews with rontline sta
Feedback rom training sessions
Feedback is not scientic. It will give you anecdotal evidence, rather than statistically
robust measures. Use as many dierent sources as possible and ensure that your analysis
refects the limitations o using such inormal methods (see Stage 4 or more detail).
$ Useful tip data protection
I youre asking members o the public to provide you with personal
inormation, this will need to be gathered and stored in line with government
and industry standards. Check your departments inormation risk policy and
talk to market research specialists in your marketing hub i you need more
advice.
Reviewing any remaining gaps
Ater gathering the data, review your evaluation plan to identiy any perormance metricsthat have no data source. I gaps still exist, consider how important it is to measure that
particular perormance metric or KPI. I it is not central to the evaluation, you may choose
not to measure it at all but point out this limitation in the nal report. I a perormance
metric or KPI is crucial to your evaluation, consider asking or additional resource or budget
to measure it, and make clear the implications o not obtaining it.
Agreeing who will collect data
Beore the activity runs, agree who will gather the data or each source that you have
identied, when and in what ormat. Collecting each piece o data in the same ormat
using consistent time periods and target audiences will help with the analysis later on.
-
7/30/2019 GCN Evaluation Guide
22/50
Evaluating Government Communication Activity
22 Stage 3: Implement
Completing the evaluation plan
You should now complete your evaluation plan by setting out the data sources that you
will use or each o your perormance metrics and by noting any gaps that still exist. Your
completed plan will now include:
objectivesandtargetaudiences
performancemetricsandKPIs,andthedatasourcesthatyouwillusetomeasureeachone
anylimitationsinyourevaluation
agreedbudgetandresourceneededtogatherthedata(signedoffbythebudget-
holder i needed).
The three Cs principles or good data collection
Whatever youre measuring, make the data that you collect as continuous, consistent
and comparable as possible.
Continuous Most communication activity aims to get people to start, stop or
continue a particular attitude or behaviour. To quantiy its eect,
try to measure peoples attitudes or behaviour beore, during and
ater the activity runs. Benchmarking beore it runs is particularly
important to demonstrate the eect o your activity. For ongoingactivity, measure perormance regularly enough to show the eect
it is having on attitudes or behaviour. The more data points you can
capture, the more obvious the trends will be.
Consistent Use consistent measures and methodology to assess activity that you
repeat or run continuously over time. Using the same wording or
questions, tracking against the same audience and collecting data
in the same way every time will enable you to measure longer-term
trends accurately.
Comparable As ar as you can, try to use the same measures or every piece
o activity that you run. Try to make your measures as similar aspossible to the ones used by other communicators in your team
and across government. This will make it easier to compare results
in the uture. The appendix gives you guidance on the types o
measures that you might want to use or your activity to help with
standardisation.
-
7/30/2019 GCN Evaluation Guide
23/50
Evaluating Government Communication Activity
Stage 4: Analyse and report 23
Stage 4: Analyse and reportperormance against the plan
Checklist
Task: to assess the success o your activity by:
Analysing eectiveness
Demonstrating eciency and value or money
Output: nal evaluation report
Analysing eectiveness
Once your activity has run, you will need to measure how eectively it met its objectives.
Begin by gathering data and evidence rom all your sources and bringing it together in a
centralised database or older. Check to see whether it looks correct beore beginning the
analysis. Also, check that the activity ran as planned and whether anything unexpected is
likely to have aected its perormance.
$Useful tip activity diaryYou will nd it useul to keep a diary while your activity is running, noting
down any external or operational actors that might aect perormance as
they happen (e.g. bad weather aecting event attendance; negative news
stories aecting public perceptions; or downtime on a website aecting
visitor numbers). This will make subsequent analysis easier.
How to approach analysis
Did your activity work as you expected?
At Stage 2, you mapped out how you expected your activity to work. Use this as the basis
or your analysis. Create some key hypotheses results or outcomes that you might expect
to see based on your objectives, activity map and past perormance.
Check perormance against your KPIs and targets
Have these been met? I so, what is driving success? I not, consider whether your targets
were realistic and whether your KPIs are accurate measures o success. For example, i you
have set KPIs or a new activity, you may need to consider revising these in the uture.
-
7/30/2019 GCN Evaluation Guide
24/50
Evaluating Government Communication Activity
24 Stage 4: Analyse and report
Be objective
Analyse the ull range o potential outcomes and results or your activity. Do not ignore
results or trends that dont t with your hypotheses or with the general patterns in the
data. Instead, look or the reasons behind these. I something didnt work in the way that
you expected or ailed to meet its objectives, look at the reasons why. This will help you to
amend uture activity and so improve uture perormance.
Try to isolate any operational or external issues
Did any unoreseen operational issues aect your activitys perormance? For example:
Werethereenoughcontactcentrestafftorespondtothecallswhichyouractivitygenerated?
Didthewebsitetowhichyouweredirectingpeoplestopworking?
Didexternalfactors(e.g.negativePR,badweather,theeconomicsituation)havean
impact on your activitys success?
An activity diary can be very useul in helping you to identiy these actors.
Analyse data on a bottom-up basis
I you are analysing the eect o more than one activity or channel, analyse data on a
bottom-up basis against your evaluation plan. First, analyse the eectiveness o each
individual activity against its sub-objective. Then, look at the eect that each activity hadon the wider communication objective, considering which one had the greatest eect.
Assess the eect o communication on the policy objective
Consider the eect that communication as a whole has had on the wider policy objective,
bearing in mind the eect o other interventions designed to meet that objective. You
may not have sucient data to ully isolate the eect o communication on the policy
objective, but try to draw conclusions based on the available evidence.
Review progress
When evaluating activity that will not achieve its communication objectives or some time,
look or evidence o progress towards these based on the KPIs and perormance metrics
that you have identied.
Cross-check your conclusions
Where you have data rom a range o dierent sources, check to see i the results rom
each one are pointing towards the same conclusions. This is particularly important where
you are relying on less robust data sources such as inormal eedback. One set o results
may not always give a conclusive answer, but several pieces o evidence all pointing in the
same direction may allow you to be more condent in your conclusions.
-
7/30/2019 GCN Evaluation Guide
25/50
Evaluating Government Communication Activity
Stage 4: Analyse and report 25
Signpost gaps and limitations
Your evaluation plan will identiy the gaps in available data and the implications o those
gaps. Be aware o the limitations that this puts on the evaluation. Where possible, try to
draw assumptions about missing results rom the available data. For example, i you dont
know how many people have stopped smoking as a result o your stop-smoking event,
can you make an assumption about possible cessation rates based on the number o
people who tell you that theyre going to quit at the end o the event? I you cant make
such assumptions, signpost missing data within the evaluation report.
Get specialist support where necessary
Some evaluation will require more sophisticated analysis, or example econometricmodelling techniques to predict uture perormance or isolate the eect o dierent
actors on perormance. This type o analysis will generally be carried out by specialists,
either within your department or externally. Only seek external support where you are
unable to carry out the analysis within your marketing hub.
Demonstrating eciency and value or money
When you evaluate perormance, always ask yoursel whether the results that you
achieved justiy the time, resource and money that you spent. As part o your evaluation,
also consider whether the results that the activity achieved justiy the time and money
that were spent on it. Where you are able to quantiy the actual eect that the activity
had on the overall policy objective and put a nancial value on this, you should be able
to calculate the overall return on marketing investment. Otherwise, calculate the cost per
result the amount o time or money that was invested or each person who carried out
a specied action. Compare your results with other activities that you or others have run
to see which ones are most cost-eective and time-ecient and use your learning to
optimise uture activity.
Examples o return on marketing investment and valid cost per result gures include:
Return on marketing investment: Where you are able to demonstrate how many lives
have been saved as a result o a campaign to reduce speeding on residential streets and
calculate the nancial value o each lie saved, the return on marketing investment is thetotal number o lives saved multiplied by the nancial value o each lie saved minus the
cost o running the campaign.
Cost per result: Where you are not able to demonstrate how many lives have been saved
as a result o the campaign, you may instead choose to calculate the cost per result, basing
this on any action that people took ater seeing the activity. This might include the cost
per web visit (total web visits divided by total campaign cost) i the activity sends people
to a website or more inormation, or the cost per event attendee (total event attendees
divided by total campaign cost) i the activity involves running events on road saety.
Further guidance on demonstrating nancial return is available in a separate GCN
publication.3
3 Evaluating the nancial impact o public sector marketing communication: An Introduction to Payback, Return on Marketing Investment (ROMI)and Cost Per Result (https://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/intro-to-payback-romi-and-cpr.pd)
-
7/30/2019 GCN Evaluation Guide
26/50
Evaluating Government Communication Activity
26 Stage 4: Analyse and report
The evaluation report
You will have agreed the ormat or your report at Stage 1. Check that this is still correct.
Whatever ormat your report is in, always try to ollow these principles:
beobjectiveandincludeallresults,positiveandnegative
alwaysprovideclearconclusionsandrecommendationswhatshouldbedoneasa
result o this report and by whom?
separatefactfromopinionandrecommendationssothatotherscanreviewthe
evidence on which you have based your conclusions
acknowledgegapsinthedataandtheirimplications
stateanyassumptionsthatyouhavemadeandwhatyoubasedtheseon
includereferencesforalldatasourcesusedinyourreport
includeappropriategraphsandimagestoaidinterpretation.
A template or your evaluation report is included here. I you are submitting an evaluation
report to ERG, you must use the ollowing template.
https://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/GCN-Evaluation-report-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/ERG-Evaluation-report-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/ERG-Evaluation-report-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/GCN-Evaluation-report-template-.doc -
7/30/2019 GCN Evaluation Guide
27/50
Evaluating Government Communication Activity
Conclusions 27
ConclusionsAdopting the principles, processes and practical examples set out in this guide will
enable you to design and implement evaluation plans or the ull range o government
communication activity to the standards required by ERG and the Communication Delivery
Board. By ollowing the recommended approach, you can also be condent that your
work will meet the revised GCN competencies or government communicators.
Adopting good evaluation standards will also enable you both to demonstrate the
contribution that your communication activity makes towards achieving overall policy
objectives and to make a well-inormed business case to invest urther resource andbudget in your work. Sharing your learning with other government communicators in your
team, department or arms-length body and hub, and across GCN more widely, will enable
others to benet rom your knowledge and insight as well.
Finally, remember that it can take time to ully implement the approach set out in this
guide and to gather the data needed to understand what is driving success. Begin
gradually, recognising that it is always better to produce a partial evaluation than nothing
at all, provided that you are clear on the limitations o your end report.
This guide and the examples that it contains refect knowledge and experience drawn
rom a wide range o colleagues rom across the government communication network.
Further comment, eedback and examples are always welcome. Please contact
Helpul contacts and resources
I you want more help or support in applying the content in this guide, talk to your hub
lead or to the evaluation specialists within your hub. The GCN website includes their
contact details.
There are also a number o teams within the Government Communication Centre who
may be able to help:
The Campaigns and Strategy Team works with the communication hubs to dene
good practice evaluation standards across government and to ensure that these are
refected in all annual communication plans and ERG submissions.
TheEvaluation Team in the Shared Communications Service can provide urther
advice on how to apply these evaluation standards to specic projects; they can
oer practical support in planning and conducting your evaluation and advice on the
procurement o external agencies and should be your rst point o contact or general
evaluation queries.
TheGovernment Procurement Service will provide access to specialist external
agencies, i required.
TheGovernment Communication Network runs training courses on evaluation and
the GCN website contains a wealth o inormation and access to groups and individuals
to help you with your evaluation. Go to https://gcn.civilservice.gov.uk/ or more
inormation.
https://gcn.civilservice.gov.uk/https://gcn.civilservice.gov.uk/https://gcn.civilservice.gov.uk/https://gcn.civilservice.gov.uk/ -
7/30/2019 GCN Evaluation Guide
28/50
Evaluating Government Communication Activity
28 Appendix
Appendix
Recommended metrics
Introduction
This appendix gives examples o the types o perormance metric that you should
consider including in your evaluation plan. They are split by discipline:
1. Press
2. Marketing
3. Internal communication
Within each discipline, perormance metrics are urther split by channel and
objective type. Choose the set that is most relevant to your activity, adapt it as
appropriate and include it in your evaluation plan. I you are using more than one
channel, you will need to pick separate sets or each channel that your activity uses.
I your activity reaches the end audience via an intermediary (e.g. the media or
a partner), use separate sets o metrics to measure:
how eectively your activity engaged the intermediary;
how eectively your intermediary engaged the end audience on your behal.
Templates or activities with and without an intermediary audience are available on
the GCN website.
$
Useful tip adapting perormance metrics to your activity
Ensure that you are clear on your activitys objectives, the channelsthat you are using and the key messages that you are promoting.
This will enable you to adapt the example perormance metrics to
t your activity.
https://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/Evaluation-plan-template-2.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/Evaluation-plan-template-2.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.doc -
7/30/2019 GCN Evaluation Guide
29/50
Appendix 29
Evaluating Government Communication Activity
1. Press activity
Press activity includes proactive publicity (activity which proactively promotes
a policy or your organisation), reactive media handling (activity put in place to
respond to a specic issue or event or to rebut inaccurate coverage o this issueor event) and media briefng and handling sessions that you organise to support
ministers and senior policy ocials in their work. All types o press activity have
an intermediary audience (e.g. the media or other spokesperson (a minister or
senior ocial)) and an end audience (generally members o the public).
Choose the perormance metrics that are most relevant or your activity rom the
list below.
$
Useful tip evaluating intermediary and end audiences
The nal outcome or your intermediary audience should always orm
the input or your end audience use this evaluation plantemplate to
help you.
Inputs or intermediary audience
The number and nature o press or media activities that you carry out. This
might include:
Proactivepublicity the number o PR activities, media briengs and packagesissued to the media
Reactivemediahandling the number o corrections, reactive statements andrebuttals issued, the number o interviews arranged
Mediabriefngandhandling the number o media briengs and mediahandling sessions that you organise or ministers and ocials
Any costs incurred in running the activity, time and internal resources used.
Outputs or intermediary audience
Proactivepublicity the number o media contacts that you reach with youractivity, the number o times that you contact them, the messages that you
pass onReactivemediahandling the number o media contacts that you reach with
your corrections, reactive statements and rebuttals, the number o interviews
that take place
Mediabriefngandhandling the number o briengs and training sessionsthat you organise, the inormation and skills that you pass on.
https://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.doc -
7/30/2019 GCN Evaluation Guide
30/50
Evaluating Government Communication Activity
30 Appendix
Out-takes or intermediary audience
Proactivepublicity the attitude o the media overall (and key media contacts
where applicable) towards the message that you are promoting or towardsyour organisation more generally
Reactivemediahandling the attitude o the media towards the issue that youare working on and your handling o it. Have you changed their knowledge
or attitude?
Mediabriefngandhandling what do ministers and senior ocials thinkabout the brieng and training that you provide? Do they nd it useul?
Do they intend to put it into practice?
Intermediate outcomes or intermediary audience
Proactivepublicity the number o media contacts that you reach with youractivity, the number o times that you contact them, the messages that you
pass on
Reactivemediahandling the number o media contacts that you reach withyour corrections, reactive statements and rebuttals, the number o interviews
that take place
Mediabriefngandhandling knowledge and skills that ministers and ocialsgain as a result o your brieng or training sessions.
Final outcomes or intermediary audience/Inputs or end audience (these
perormance metrics are the same)
The volume and quality o media coverage achieved by your activity. This might
include:
Proactivepublicity number o pieces o coverage achieved, accuracy ocoverage, avourability o coverage, key message penetration, quotes and
interviews used
Reactivemediahandling number o pieces o coverage or interviewscontaining your responses, amendments or corrections, overall accuracy o
coverage, avourability o coverage, the amount o negative coverage that has
been prevented as a result o your work
$Useful tip measuring a reduction in negative coverage
It is oten ar harder to measure the negative coverage that has
been avoided as a result o the corrections and rebuttals that
you issue. By keeping a record o the contacts that you have with
the media or each issue that you work on, over time it becomes
easier to demonstrate how your interventions are aecting
coverage in the longer term.
Mediabriefngandhandling the number o times that your contacts use theknowledge and skills that you have passed on in their contact with the media,
the volume o coverage achieved as a result o these contacts, accuracy and
avourability o coverage, key message penetration.
-
7/30/2019 GCN Evaluation Guide
31/50
Appendix 31
Evaluating Government Communication Activity
Outputs or end audience
The number or percentage o your end audience reached by the media activity
and how oten they saw it.
Out-takes or end audience
Use the perormance metrics included in section 2.1 (purchased marketing
activity) to measure what your end audience recall, think or eel about the
media activity.
Intermediate outcomes or end audience
Depending on the nature o the messages that you promote, you may expect
your end audience to take some orm o action as a result o seeing the media
coverage. This might include:
Seeking inormation rom you
Seeking inormation rom other sources
Registering or a service, product or inormation
Starting, stopping or continuing a particular behaviour.
Use the perormance metrics included in section 2.1 (purchased marketing
activity) to measure each o these actions where relevant.
Final outcomes or end audience
Final outcome measures should assess whether your activity met its overall
communication objective, and its eect on the overall policy or reputation
objective that you are working to and the eect that this has had. Choose
perormance metrics that enable you to measure this.
-
7/30/2019 GCN Evaluation Guide
32/50
Evaluating Government Communication Activity
32 Appendix
2. Marketing activity
Choose the perormance metrics that are most relevant rom the ollowing list,
based on the type o activity that you are evaluating:
2.1 Purchased media
2.2 Websites, social media and other digital spaces
2.3 Public and stakeholder engagement
2.4 Partnership activity(including public advocacy).
2.1 Purchased media (TV, radio, press, outdoor, digital advertising, paid-or
search, direct marketing)
$Useful tip evaluating multi-channel activityI your activity includes more than one paid-or channel, remember to
include perormance metrics to measure the eectiveness o each one.
Inputs
The number o people you plan to reach with your activity and requency o
exposure. This might include:
Estimated reach, coverage and requency (TV, press, radio, outdoor)
The number o impressions served (digital advertising)
The number o planned clicks (paid-or search)
The number o inserts produced, the number o leaets planned to be
distributed (direct marketing)
The costs (media and production) incurred (by channel).
Outputs
The number o people actually reached by your activity and the number o
times they were exposed to your message.
$Useful tip sourcing data
I an agency is booking media or organising distribution o direct
marketing materials, it will be able to provide input and output data
or you.
-
7/30/2019 GCN Evaluation Guide
33/50
Appendix 33
Evaluating Government Communication Activity
Out-takes
Think about the key messages that you want your activity to get across to your
end audience what will they be thinking or eeling i its been successul? Adaptthe ollowing perormance metrics to enable you to measure their reaction,
choosing those which are most relevant or your activity.
$Useful tip identiying key messages
Use your activity map to help you to identiy what your key
messages are and how you expect your target audience(s) to
react to them.
Have they seen it (RECALL)?
Are the end audience aware o the key/supporting message that you are
promoting?
Have they seen any advertising, communication or publicity about the key/
supporting message?
Can they describe this advertising, communication or publicity? Where have
they seen the advertising, communication or publicity?
Do they recognise your activity when its shown to them (this might include
advertising, leaets, letters, event invitations etc produced by you or press
coverage and partnership activity delivered via an intermediary)?
Do they understand the activitys key messages (THINK)?
Did the end audience like the activity?
Did they think it was relevant to them?
Was it clear and engaging?
Did it capture their attention?
Did they understand the key/supporting message that the activity was
promoting?
Has it impacted on their views? Do they intend to do anything as a result (FEEL)?What would you want the end audience to eel or think i the activity was
successul? On this basis, what eect did the activity have on your end/
intermediary audiences attitudes?
Do the end/intermediary audience eel more positive about your department
and its work as a result o the activity? Are they willing to support you in your
work or to advocate your department more widely?
Do the end/intermediary audience intend to take the action specied in the
key/supporting message in the coming weeks or months (depending on the
time-rame or your activity)?
-
7/30/2019 GCN Evaluation Guide
34/50
Evaluating Government Communication Activity
34 Appendix
Intermediate outcomes
Intermediate outcome metrics measure the actions that people have taken
as a result o your activity. These will vary depending on your communicationactivity, but might include:
a) Seeking inormation rom you
b) Seeking inormation rom other sources
c) Registering or a service, product or inormation
d) Starting, stopping or continuing a particular behaviour
e) Interacting with you or with others online or ofine.
Suggested perormance metrics or each o these intermediate outcomes aregiven below choose the one(s) that is/are most appropriate or your activity
and adapt as appropriate.
a) Seeking inormation rom you
Sometimes, you will ask people to contact you directly or more inormation.
Always try to include a range o metrics that cover who contacted you, how
they contacted you and the depth o contact. Even i you dont directly ask
people to contact you, its always worth checking to see whether youve had
uplit in contact as a result o your activity. Suggested perormance metrics or
the dierent contact channels include:
Call centre:
The number o people calling
What drove them to call (your activity, activity carried out by
intermediaries, e.g. media, stakeholders, other actors)
The reason or their call
How many callers you passed relevant inormation on to
Digital activity (advertising):
The number o people clicking on digital advertising or text links
(split by creative, site)
The number visiting your website or social media space as a result o
advertising
Website or social media space (NB: data or these metrics will be
available rom the Government Digital Service (GDS)):
Source o visits (how people came to the site)
Average length o time spent on the site (and number spending more
than 30 seconds on the site)
The number o pages visited
The number watching any videos or embedded content on your site
(starting to watch and watching whole clip)
The number seeking relevant inormation
Bounce rate (percentage only visiting one page)
-
7/30/2019 GCN Evaluation Guide
35/50
Appendix 35
Evaluating Government Communication Activity
Intermediate outcomes (continued)
a) Seeking inormation rom you (continued)
Events/ace-to-ace The number attending events as a result o activity you ran to promote them
The number visiting you or your sta in person at other times
What drove them to visit (your activity, other actors)
How many you passed relevant inormation on to
Correspondence
Number o emails received (positive/negative)
Number o letters received (positive/negative).
b) Seeking inormation rom other sources
I your activity is being delivered by stakeholders or partners, ask them whether
they can gather data on how people seek inormation using the perormance
metrics listed under section (a) above.
Other sources that you might want to include are:
People searching or inormation on your activity online
People who claim to have asked riends, amilies or other organisations or
inormation or advice ollowing your activity.
c) Registering or a service, product or inormation
Your activity might ask people to register or a service, more inormation or a
product, or to attend an appointment with you or a stakeholder. I so, you might
want to include measures that look at:
The number o people seeking inormation on the service, product or
inormation eatured in your activity (see previous section)
The number o people requesting a product eatured in your activity (and
where they heard about this product)
The number who actually receive this product
The number who go on to use this product
The number o people signing up to receive inormation in the uture
The number o people starting the registration process or a service or
product (and where they heard about it)
The number o people completing the registration process
The people using the service that they have registered or and how they use it
-
7/30/2019 GCN Evaluation Guide
36/50
Evaluating Government Communication Activity
36 Appendix
Intermediate outcomes (continued)
c) Registering or a service, product or inormation (continued)
$Useful tip choosing the right perormance metrics oryour activity
The perormance metrics that you choose here will vary
depending on the nature o your activity. For example, i youre
running an ongoing programme to support people as they give
up smoking, you might record how many people attend an initial
meeting or read the inormation that you have sent to them
and then record how many attend subsequent meetings,
request urther inormation etc. Creating an activity map can
be a helpul way o identiying how you would expect people
to use the service, and can help you to choose the rightperormance metrics.
The number o people making an appointment with you or a stakeholder
The number o people attending this appointment.
d) Starting, stopping or continuing a particular behaviour
Your activity may ask people to start, stop or continue a particular behaviour.
I so, you might want to include perormance metrics that look at:
The number o people undertaking that behaviour beore the activity ran
The number o people who take the relevant steps to change their behaviouras a result o your activity
The number o people who continue with the changed behaviour over time.
e) Interacting with you or with others online or ofine
You may want people to pass your message on or to discuss it with others.
I youre using social media, you may want them to join an online community
or contribute to content on your website.
You may nd the ollowing metrics useul, depending on what media youre using:
The number o people who pass your message on or discuss your activity orthe message that it is promoting with others
The number o people who claim to have discussed your message with
others (online and ofine)
The number passing on or sharing your content digitally (e.g. tweets,
retweets, sharing link on Facebook, via Digg, Delicious, StumbleUpon,
Reddit etc)
Amount o talk on digital orums, blogs, websites etc (tracked via buzz
monitoring)
Amount o this talk that is positive (and accurate)
-
7/30/2019 GCN Evaluation Guide
37/50
Appendix 37
Evaluating Government Communication Activity
Intermediate outcomes (continued)
e) Interacting with you or with others online or ofine (continued)
The number o people who interact with you online depending on thedigital space that you use, this might include:
The number o Facebook ans or riends, the number liking your content
The number o Facebook ans or riends who are active (i.e. return to the
site regularly)
The number o people posting content on your site
The number o people commenting on your content (e.g. on your own
site, Facebook, YouTube).
$
Useful tip passing the message on
I your activity asks someone to pass a message on on yourbehal, also include metrics that enable you to measure the eect
o this message on those to whom they speak.
Final outcomes
Final outcome measures should assess whether your activity met its overall
communication objective, and its eect on the overall policy objective that
you are working to and the eect that this has had. Choose perormance
metrics that enable you to measure the activitys impact on both.
2.2 Websites, social media and other digital spacesGDS will produce standard reporting dashboards or gov.uk. These include
standard perormance metrics that measure outputs, out-takes and intermediate
outcomes. Bespoke reports or gov.uk and other government websites and social
media spaces will also be available. Further inormation on how to get this data will
be available soon via the GCN website. Always ensure that you integrate these
perormance metrics with wider measures that look at the eect o your website
or social media activity. These might include:
Inputs
The website, social media or other digital space and related content created
to meet communication objectives
The cost and resource used to create this website, social media space or
content.
Outputs (available via GDS reports)
GDS can provide perormance metrics that you can use to look at how many
people were exposed to your digital content. These will include:
The number o the end audience visiting the site or digital space at least once
during the evaluation period (unique users)
The source o visitor reerrals.
-
7/30/2019 GCN Evaluation Guide
38/50
Evaluating Government Communication Activity
38 Appendix
Out-takes (available via GDS reports)
What people think, eel or recall about the webspace. Section 2.1 gives more
detail on the types o perormance metric that you might want to include.You may also be able to iner attitudes and understanding rom comments
that people leave on your website or social media space.
Intermediate outcomes (available via GDS reports)
GDS can provide a number o standard metrics that measure how people
interact with your site or digital space. These will include:
Average length o time spent on site
Bounce rate
The number watching any videos or embedded content on your site (starting
to watch and watching whole clip)
The number o pages visited.
Also consider including metrics that look at actions carried out on social media
websites, such as:
Number o likes on Facebook
Number o retweets on Twitter.
Always ensure that you tie these standard metrics back to your website or
digital spaces objectives and consider what you are trying to get people to
do (register, read a particular piece o content, retweet content etc). This will
enable you to interpret the data appropriately.
Final outcomes
Final outcome measures should assess whether your website or digital space
met its overall communication objective, and its eect on the overall policy
objective that you are working to and the eect that this has had. Choose
perormance metrics that enable you to measure the activitys impact on both.
2.3 Public and stakeholder engagement
Public or stakeholder engagement activity is put in place to secure eedback on orsupport or a specic policy area. Depending on the activity that you are running,
you may use a range o channels including events, digital and purchased channels.
Inputs
The activity that you carry out to promote your initiative. Depending on the
nature o your engagement activity, this might include:
The number o invitations to events that you send out
The number o requests to provide eedback that you issue
Activity that you carry out to publicise your engagement activity
The website that you create to gather eedback
The number and nature o contacts that you make with specic
stakeholders.
-
7/30/2019 GCN Evaluation Guide
39/50
Appendix 39
Evaluating Government Communication Activity
Outputs
Depending on the nature o your engagement activity, these might include:
The number o people that you reach via your publicity, invitations andrequests
The number o people attending the event that you organised
The number o people who visit your website or digital space (see
section 2.2 or more detail on digital metrics)
The number o conversations that you have with a specic stakeholder as
a result o the contacts that you make.
Out-takes
Think about what you want the audience that you are engaging to recall, think
or eel as a result o your engagement activity. Oten there will be a specicpolicy that you want them to advocate or support. Include relevant out-take
measures to assess levels o support or your message and your department.
Use the perormance metrics included in section 2.1 (purchased marketing
activity) to help to measure the depth o engagement and attitudes towards
your policy area.
Intermediate outcomes
Intermediate outcome metrics measure the actions people have taken as
a result o your activity. These will vary depending on your communication
activity, but typical intermediate outcomes or stakeholder and public
engagement include:
a) Providing usable eedback
b) Attending an event
c) Passing your message on to others.
Suggested perormance metrics or each o these intermediate outcomes are
given below choose the one(s) that is/are most appropriate or your activity
and adapt as appropriate.
a) Providing usable eedback
You may be running activity to increase engagement among specic audiences
possibly to seek help in developing policy or to ask or eedback or consult on a
particular policy area. I so, you might include measures that look at:
The number o people who respond to your engagement activity in any way.
This might include:
Comments or eedback received on your website, in writing, by phone,
ace-to-ace
The quality o eedback and comments received
The number o people attending public meetings or hearings
The number o people agreeing to help to develop a specic policy area
(generally in relation to stakeholder engagement)
The amount o usable eedback incorporated into uture policy development
or engagement activity.
-
7/30/2019 GCN Evaluation Guide
40/50
Evaluating Government Communication Activity
40 Appendix
Intermediate outcomes (continued)
b) Attending an event
You may invite stakeholders or members o the public to attend events so thatyou can share inormation or increase engagement. Depending on what your
event is designed to do, you might want to think about including perormance
metrics that look at:
The number o event attendees who interact with you or your sta during
the event
The number o event attendees who undertake specic actions or
behaviours in line with messages promoted at the event
The number who thought the event was good and useul
The number who understand the messages that you were promoting.
c) Passing your message on to others
You may want people to pass on specic messages rom your engagement
activity. You may nd the ollowing metrics useul, depending on what media
youre using:
The number o stakeholders or members o the public who pass your message
on and discuss your activity or the message that it is promoting with others
The number o stakeholders or members o the public who claim to have
discussed your message with others (online and ofine):
The number passing on or sharing your content digitally (e.g. tweets,
retweets, sharing link on Facebook, via Digg, Delicious, StumbleUpon,
Reddit etc)
Amount o digital talk on orums, blogs, websites etc (tracked via buzz
monitoring)
Amount o positive (and accurate) digital talk.
$Useful tip passing the message on
I your activity asks someone to pass a message on on your
behal, also include metrics that enable you to measure the eect
o this message on those to whom they speak.
-
7/30/2019 GCN Evaluation Guide
41/50
Appendix 41
Evaluating Government Communication Activity
2.4 Partnership activity (including public advocacy)
In partnership activity a partner is used as an intermediary to advocate your
work, pass on a message or deliver a service to an end audience on your behal
typically in the areas o policy delivery and reputation management. Partnersmay include commercial, not-or-prot and government bodies as well as
individual stakeholders.
$Useful tip evaluating partnership activity
All partnership activity will have an intermediary audience the partner
that you are engaging. Always think about who your end audience is and
what you are asking the partner to do to reach them on your behal.
Inputs or intermediary audience
Details o your overall partnership engagement strategy and the supportingactivity that you carry out to engage stakeholders or partners so that they will
pass on your message on your behal. This might include:
The number o partners that you contact by phone, email or ace-to-ace
The number o partners that you invite to events, training etc and the
number o events and training sessions that are held
The number o times that you contact each partner (particularly when
building relationships and hence advocacy over time)
The number and nature o key messages that you pass on to each partner(particularly when building relationships/advocacy)
Any costs incurred in contacting partners, time and internal resources used.
Outputs or intermediary audience
The number o partners who are exposed to the activities that you run and the
messages that you promote. This might include:
The number o partners successully reached by your activity (the number
with whom you are able to share your messages)
The number o partners attending events or training that you have organised
The quality and range o messages shared with each partner over time
(particularly when building advocacy).
Out-takes or intermediary audiences
Think about what you want partners to think or eel as a result o the contact
that you make. This might include:
The number o partners who support your activity
The number o partners who are interested in nding out more about your
initiative or in working with you.
-
7/30/2019 GCN Evaluation Guide
42/50
Evaluating Government Communication Activity
42 Appendix
Intermediate outcomes or intermediary audiences
Intermediate outcomes will depend on the nature o contact that you make with
partners and the response that you are seeking rom them. They might include:
The number o partners contacting you or more inormation
The number o partners discussing the activity with you
The number o partners expressing an interest in working with you
The number o partners agreeing to undertake a specic activity or pass on a
specic message or you.
Final outcomes or intermediary/Inputs or end audience
The number and/or nature o messages passed on or activities carried out to
promote messages on your behal by partners. This might include:
The number o partners advocating the work o your department or team:
The number o partners passing on specic messages about your
department and its work (through conversations with others, interviews,
media coverage, digital interaction etc)
The number o partners oering more general support or your department
The number o messages passed on by each individual partner, content o
each message, avourability o each message
The number passing on or sharing your content digitally (e.g. tweets, retweets,
sharing link on Facebook, via Digg, Delicious, StumbleUpon, Reddit etc)
The number o partners carrying out activities to prom