Measuring Impact

What is Impact?

Dictionary definition

1. The striking of one body against another; collision.

2. The force or impetus transmitted by a collision

3. The effect or impression of one thing on another: still

gauging the impact of automation on the lives of factory

workers

4. The power of making a strong, immediate impression: a

speech that lacked impactwww.thefreedictionary.com

Where do I start?

Work backwards

Does your coaching programme / project have an impact on coaches?

Define the impact. What impact do you hope to see?

What are the programme aims and objectives?

The importance of planning!

The Planning Stage...Project Logic Models

Rationale/ Aims & ObjectivesAll coaches are sufficiently skilled and qualified to provide

the best experience to participants. In turn increasing recruitment and retention

Challenges

Any barriers?What might you

encounter?

Opportunities

Where can you help?

What can be done to overcome challenges?

Inputs

£Resource

AdminEtc.

Activities

CPD courses / Qualifications / Educational

awareness raising etc.

Outputs

More coaches completing

Level 2 qualification

Outcomes (short and

medium term)

Coaches feel better skilled,

knowledgeable and more confident

Impact (long term)

Coaches provide better

experiences for

participants = increased

recruitment and retention

Output Measures

Increase in % of those completing Level 2

(database)

Outcome Measures

Increase in % of coaches feeling x,y,z (Coach

questionnaire)

Impact MeasuresObservational

assessment, participant feedback, tracking

survey

The Planning Stage...Project Logic Models

Rationale/ Aims & ObjectivesAll coaches are sufficiently skilled and qualified to provide

the best experience to participants. In turn increasing recruitment and retention

Challenges

Any barriers?What might you

encounter?

Opportunities

Where can you help?

What can be done to overcome challenges?

Inputs

£Resource

AdminEtc.

Activities

CPD courses / Qualifications / Educational

awareness raising etc.

Outputs

More coaches completing

Level 2 qualification

Outcomes (short and

medium term)

Coaches feel better skilled,

knowledgeable and more confident

Impact (long term)

Coaches provide better

experiences for

participants = increased

recruitment and retention

Output Measures

Increase in % of those completing Level 2

(database)

Outcome Measures

Increase in % of coaches feeling x,y,z (Coach

questionnaire)

Impact MeasuresObservational

assessment, participant feedback, tracking

survey

Measures

The Planning Stage...methodology & data collection

Costs / Resources

Aims & Objectives

Project Outline

Project Sample

Data sources and Collection

Stakeholders / data holders

Timings and Deadlines

Title

Reporting

Quantitative - Numbers

Example; Evaluation Surveys

Project Start

Aims & Objectives

Project End

Evaluation & Conclusions

Evaluation survey

Do coaches report feeling more satisfied / confident / knowledgeable

etc. as a result of the project?

Ok, but nothing to compare the data to. Relying upon people to remember how they felt at the

start

?

Example; Tracking Studies

Has the project resulted in the desired changes in attitudes, opinions, reported behaviours etc.?

Project Start

Aims & Objectives

Baseline survey

Project End

Mid-point survey

Final survey

Look for any changes in responses Look for any changes in responses

Look for any changes in responses

Evaluation & Conclusions

Qualitative - Words

Example; Interviews / Focus groups

95% of coaches feel more confident

But in what ways? Give me examples?

Quantitative is the ‘what’

Qualitative is the ‘why’

Understanding the meaning behind the numbers

Example; Case studies

It’s about telling a...

1. Beginning

Set the scene; what was the problem

2. Middle

What happened; what was the solution / project / activities

3. End

What was the result; was the solution successful?

Case study examples

Example; Video Case Studies

Content and

Format

Script and

Set-up

Capturing on Camera

Review and Edit

Presentation and

Promotion

Excellent Coaching Case Study

Take One

29th January 2013

Researching

The FA Tesco Skills Programme

Presented by Lucy Roberts and David Hollinshead

The FA Tesco Skills programme...

•Launched in 2007 through a unique partnership with Tesco and Sport England in response to the

‘skills gap’ identified between English youngsters and their European counterparts.

•Based on a unique FA coaching model that encompasses four key attributes vital for any developing

young player: Physical, Technical, Psychological and Social elements.

•Delivers high quality football coaching for 5-11s by creating learning environments that are safe and

inclusive and coaching sessions that are simple, game related & enjoyable.

•Provides after-school football courses, holiday

coaching and specialist coaching and support in

primary schools.

•An inclusive coaching programme for 5-11s -

coaching is free of charge.

Programme objectives

•To develop better and more technically gifted players and to give every child the opportunity to be

the best that they can be.

•To provide all 5-11s with a skill set to develop their sporting ability with the long term objective of

improving the technical quality at all levels of the English game.

•Tesco objectives – to give every child in England the opportunity to get active and lead healthier

happier lives.

Achievements so far•3.98 million child places filled (42.5% girls and 57.5% boys)

•3% of places taken up by disabled children and 16.2% by ethnic minority children

•7,400 Primary schools, 569,800 school children, 22,200 teachers supported

•2,700 Charter Standard football clubs and 4,300 Charter Standard teams supported

•4,300 team managers/coaches and 10,800 coaches supported

•300 weekly Skills Centre sessions involving 4,500 children

•To demonstrate impact, research is key to everything we do

What research have we done so far?

20

0 P

aren

ts

20

Tea

cher

s

10

Co

ach

es

July 2011

30

0 P

aren

ts

15

0

Teac

her

s

10

0

Co

ach

es

November 2011

330respondents

550respondents

August 2012

40

0 P

aren

ts

in s

chem

e/

left

sch

eme

40

0 P

aren

ts

no

t in

sc

hem

e

800respondents

How long does a project like this take?

Timing Task Outcome

Week 1 Internal meeting Define scope of project: objectives and outcomes and outputs needed

Week 2 Research brief written and signed off Objectives, target audience, approach and budget

Week 3 Brief distributed 2-3 agencies

Week 4 Proposal submitted Agency selected

Week 5 Agency commissioned and set up meeting Define roles, timings and logistics

Week 6 Planning Questionnaire written, sample sourced

Week 7

Fieldwork Interviews take placeWeek 8

Week 9

Week 10 Presentation of results

Questions to include when measuring the impact of an intervention

59%

64%

67%

79%

82%

32%

27%

27%

18%

16%

High quality of coaching and supervision

I can see myself attending for many …

I will attend on a regular basis

The kickabouts are really good fun

I would recommend to friends and relatives

Strongly agree Slightly agree

98%

97%

94%

91%

91%

Encouraging children to play sport now...

...and in the future

Research shows that the FA Tesco Skills programme is

successful in...

Key Findings

Encouraging confidence in playing sport

Encouraging enjoyment in playing sport

Do you think your child would have taken part in sporting activity if the programme hadn’t been available?

41% of parents stated their kids would have doneless or no sport without the programme

Parents

20%

21%

59%

Yes would have found something else

Yes but not as much

No

Parents: Q5e (100) Would your child have taken part in activity if programme not available

“it gives children the chance to join a club and there aren’t many available in this area. If it wasn’t for this, there would be like 60 children out of it doing nothing

each week”

41%

The FA Tesco Skills is successful at

encouraging children to play sport more

frequently – especially once they have

left the programme

5-11 year olds 12-16 year olds

In FA Skills

mean 5.0 times per wk

Not in FA Skills –Control areas

mean 4.7 times per wk

In FA Skills

mean 5.2 times per wk

Not in FA Skills –Control areas

mean 3.9 times per wk

46 4352

26

4746

43

56

67

3

10

6 211

Less often

Once week

Several times a week

Daily

Q: Frequency of participating in sport

... and for longer

5-11 year olds 12-16 year olds

In FA Skills

mean 7.8 hours

per wk

Not in FA Skills –Control areas

mean 7.2 hours

per wk

In FA Skills

mean 10.2 hours

per wk

Not in FA Skills -Control areas

mean 7.0 hours

per wk

67

8

12 1

2

3

45

9

1011

67

8

12 1

2

3

45

9

1011

67

8

12 1

2

3

45

9

1011

67

8

12 1

2

3

45

9

1011

Kids currently or

previously in FA Skills

are significantly more

confident playing sport,

like sport more and are

more likely to play

sport in the future

Unweighted base: Children in FA Skills Scheme areas - 5-11 year olds in Scheme (200); not in Scheme and in control areas (200); 12-16 year olds in Scheme (107); not in Scheme and in control areas (200)

In FA Skills

Scheme

Not in FA

Skills –Control areas

In FA Skills

Scheme

Not in FA

Skills –Control areas

% agreeing5-11 yrs

5-11yrs

12-16 yrs

12-16 yrs

“Likes sport a lot”

96% 83% 94% 75%

“Confident playing sports”

71% 55% 77% 56%

“Extremely likely to take part in sport

in future”

80% 56% 79% 45%

Impact on children

•78% of parents noticed their child being more active.

•97% of parents, teachers and coaches felt that the sessions were enjoyed a lot.

•93% of parents and 88% of teachers noticed improved confidence in the children.

•77% of coaches and 51% of parents noticed improved concentration in the children.

•43% of teachers felt the children were performing better in other lessons (attributed to the social

interaction in the sessions and improved concentration and focus).

•.

Impact on teachers

•99% of teachers are more confident in teaching sport.

•97% of teachers and coaches have or plan to use the techniques taught by Skills coaches, 97% of

whom, said it was easy to use/carry them on.

•43% of teachers noticed children were behaving better in the classroom.

•92% of teachers feel that it is very important that the programme is continued to be available.•..

•/,

“Because we are a primary school, the skills of

the teachers varies considerably so the input

from professionals has increased my

knowledge. I therefore have been able to teach

differently. I also think it makes valuable links

with the people outside of school and provides

good role models.”

Research benefits

•Evidence to demonstrate that the programme is meeting its objectives to develop children, coaches

and teachers.

•Provides confidence that the programme is doing the right things to meet its objectives.

•Helps us to shape the future direction of the programme.

•Provides our sponsors with a valuable asset by making a positive impact on communities.

•Potentially will help to secure sponsor and partner investment (Tesco & Sport England).

... but don’t take our word for it...

Communicating impact - The FA Tesco Skills video

The FA Group

TheFA.com