Measuring Effectiveness

John PyneConsumer Information Department15 December 2005

Measuring effectiveness

• Why should we measure?

• What should we measure?

• How should we go about it?

Food For Thought?

“it is difficult to come to any other conclusion except that the achievement of the objectives which have been set for the FSA are non-operational in the sense that no measurement of success can be achieved”

Professor Charles Goodhart, London School of Economics.

Measuring effectivness – WHY?

• Indicates progress against objectives• Tells you what is working, what is not • Informs strategic planning and resource allocation• Focuses on value for money with limited resources • Meets stakeholder expectations, as eventually some or

all of the following will demand proof of effectiveness:– Consumers– Industry– Government– Media

Effectivness – WHAT TO MEASURE?

Measure Inputs – are we doing things right?

Information resources• Publications• Website

Communication channels • Consumer Helpline• Information centre• Website• Advertising and media

Measure Outputs – arewe doing the right things?

• What is a ‘better informed consumer? Are Irish consumers better informed now than before?

• Do they know the costs, risks and benefits of the products available?

• Is the market safer and fairer than it was before?

Measuring effectivness – HOW ?

Inputs

• Examine the main

activities you undertake

• Decide the key performance

indicators

• Measure achievement

against these

Outputs

• More complex and difficult to

measure long term goals

• Do we influence consumer behaviour?

• How do we separate our influence

from other ‘influencers’ - TV, press

and other media, government,

consumer groups, lobbyists.

Measuring effectiveness - our activities

What we measured • Awareness levels

• Satisfaction with helpline

• Website evaluations

• Publications evaluations

Measuring Awareness of the Financial Regulator -overall increase in the year to

July 05 is up from 10% to 30%

Level of awareness

0%

5%

10%

15%

20%

25%

30%

35%

Quantitative assessment of the Lo-Call centre’s performance

• Satisfaction with the Lo-call line is very high• Callers are disproportionatly aged 35-64 years• ¾ of queries are claimed as resolved on first call• 9/10 will use again and/or recommended to a friend

“it wasn’t full of jargon, they didn’t use

terminology that is only used in the

financial services business”

“it was actually a joy because so many people now have such horrible telephone manners and

you're talking to machines and its press 1 for option this and then

you end up getting cut off so it was a great relief to be connected to a human

straight off”

Measuring effectivness of communication channels – website

• Qualitative research – focus groups & one-to-one accompanied surfs.

• Task-based assignments to assess the website in terms of:

• Design

• Content

• Functionality

• Exploration of comparative websites – Expectations vs. reality

Website Evaluation - Findings

“It is just bland. It just looks dull and

boring.”

“It is just a real black and white

website. There are no pictures, nothing visual to hold your

attention.”

Initial Impact• Very little ‘excitement’ about the website• Criticised for looking a bit ‘dull’• A number claimed it is a bit ‘boring looking’• Recognition that this is a ‘serious’ subject• Some argue serious doesn’t have to mean

boring• Calls for more pictures and images on the

site• Most want a bit more ‘life’

Other Websites

• The site ‘competes’ with other websites for ‘entertainment value’.

• It is easy to leave the site because ‘first look’ does not engage the reader.

Publications evaluation - findings

INSTITUTIONAL ‘feel’

• Little sense of ‘independence’

• Likely to be confused with bank brochure

• Old fashioned – ‘looks like school text book’

• “I thought they were trying to sell me a mortgage. I could imagine picking that up in the bank and it would really be just promoting their products”

“PICK ME UP” VALUE

• Nothing new at a first glance

• Dull use of colours, looks boring

• Few claimed they would be tempted to pick up

Measuring effectiveness - our long term goals

What are our goals?

• Better informed consumers

• Aware of costs, risks, benefits

• Safe and fair market

What do we expect to see if our goals are met?

Better informed consumers

What attitudes, skills and buyer behaviour might indicate consumers are ‘better informed?’

•   they seek appropriate information and shop around before choosing a product or service;

•  they know what products they need, at appropriate times in their lives

•    they budget for things and plan ahead

•    they are not usually persuaded to buy, but proactively choose products to suit their circumstances

•    they know how to seek redress if things go wrong;

Some possibilities?

Awareness of costs, risks and benefits

What attitudes, skills and buyer behaviour might indicate consumers know more about the costs, risks and benefits?

•    they know the relationship between risk and return

•    they know the status of financial advisors and the implications of ‘tied’ vs. ‘independent’

•    they know how to compare similar products

•    they recognise when they need advice

•    they ask questions, and don’t rely completely on the advice of any party

Some possibilities?

A safe and fair market

What indicators can we use to assess how safe or fair the market is?

•    Product transparency – degree to which warnings, penalties etc. are signalled ‘up-front’ clearly and simply

•    Degree of firms’ compliance with Codes of Conduct

•    Sanctions imposed on non-compliant firms

•   Competition – number of firms, product competition, price variation, degree of product differentiation, comparative advertising by firms.

  

Some possibilities

Measuring effectiveness - HOW

Longitudinal surveys – perhaps every five years

plus

Analysis of industry data

• Have to measure actual buyer behaviour as well as knowledge, attitudes and skills

• Must allow for fact that there is no ‘ideal’ of the informed consumer.

•Must allow for fact that consumers are not a homogenous group.

Methodology?

Issues?

Consumer Protection

Irish Financial Regulator in in the process of finalising a unified Code regulating the Conduct of Business of:

• Banks, • Insurance Companies, • Financial Intermediaries, • Investment Managers,• and Money-Lenders. The Code will also be extended to cover Stockbrokers and

Credit Unions.

Question: How can we ensure that effectiveness is a key criterion in the Code development process?

Ways of Considering Effectiveness of Codes

• Consultation with Industry,

• Regulatory Impact Analysis,

• International Best Practice.

Consultation with Industry

Benefits:• Knowledge of markets,• Interest in reputation of the Industry,• Knowledge of Impacts, particularly Costs.

Risks:• Regulatory Capture.

Regulatory Impact Analysis (RIA)

• Irish Government Requires All new Regulatory initiatives to be subjected to RIA.

• The Financial Regulator has just published its RIA in relation to the Code of Conduct : www.financialregulator.ie

• Key Elements: Costs and Benefits Impact on Competition Impact on Smaller Firms

Returning to Professor Goodhart:

• His conclusion was that one of the most valuable ways of considering the effectiveness of financial regulators was peer review.

• This supports the view that the Financial Consumer Education and Protection Forum is a very worthwhile initiative, and one that we should seek to exploit to its maximum potential.

Thank youThank you