evaluating a financial literacy program: the case of the australian moneyminded program rob brooks...

Evaluating a Financial Literacy

Program: the Case of the

Australian MoneyMinded Program

Rob Brooks

Monash University

Presentation Structure Background / Context

Objectives

Overview of evaluation framework

Evaluation of the MoneyMinded program in the context of the framework

Conclusion

Background / Context

• The promotion of greater levels of financial inclusion is now seen to be a major public policy and corporate initiative across a variety of countries

• Core elements of programs to promote greater financial inclusion include matched savings programs and financial literacy education


• United States: American Dream Demonstration • United Kingdom: Saving Gateway• Australia: Saver Plus• Need for effective evaluation of programs in their

role in improving financial inclusion• Fox, Bartholomae and Lee (2005)• To date, no single framework to guide the

evaluation process


• Fox, Bartholomae and Lee extend the framework of Jacobs (1988) to develop a 5 tiered evaluation framework against which evaluations of financial literacy education programs can be conducted

Objectives• To explore this evaluation framework in the context of a

financial literacy program designed and developed in a different country, specifically the MoneyMinded financial literacy program, offered in Australia

• Extension of work of Fox, Bartholomae and Lee (2005)

• Provides information on the generalisability of framework

• Results will be of interest to a range of countries given comparable initiatives relating to financial inclusion across various countries

The Evaluation Framework

5 tiers, based on the work of Jacobs (1988)

1. Pre-implementation

2. Accountability

3. Program clarification

4. Progress towards objectives

5. Program Impact

Pre-implementation

Involves a needs assessment on the financial literacy training required by participants

• Needs identified from more general financial literacy testing

• Specific needs assessment around program participants

Accountability

Involves the collection of data on the education and services provided as part of the program, the costs involved and demographic information on participants

Program Clarification

Involves ongoing assessment of the program's strengths and weaknesses, and plans to review the program in the light of such assessments

Progress Towards Objectives

Involves the collection of data on the impacts of the program on participants

Program Impact

Builds on the previous stage in terms of long and short term impacts, typically involving comparison of participants and non-participants

The MoneyMinded Program General objective: To assist people to make informed

financial decisions and take control of their finances for their future

6 topics separated into 17 workshops*• Planning and Saving

• Easy Payments

• Understanding Paperwork

• Living with Debt

• Everyday Banking and Financial Products

• Rights and Responsibilities*(original version which has now been revised)

The MoneyMinded Program Allows individual participants to choose which workshops

they wish to complete Developed in part as a response to the results from the

first-ever major survey of consumer financial literacy in Australia (ANZ, 2003)

Results of the survey identified the following:• The groups within the community that could benefit

from improved financial literacy education • The issues that could be addressed in such an

education program Involvement of community groups in the delivery of

program

The MoneyMinded Program

ANZ plans to partner with around 100 community organisations during the next 5 years to deliver the program to 100,000 people nationally

Evaluation is critical for a program as comprehensive and extensive as MoneyMinded

Importance of evaluation being collected and analysed in a systematic manner

Evaluating the Money Minded Program

1. Pre-implementation needs assessment

• Not a formal needs analysis at the individual participant level

• Based on the results of the financial literacy survey reported in ANZ (2003)

• Telephone survey customised to particular needs and circumstances

Pre-implementation Stage (Cont’d)

ANZ survey identified the following groups were strongly over-represented in the lowest financial literacy quintile:

• Lowest education levels• Not working or in unskilled work• Lower income levels• Lower savings levels• Single• Extreme ends of the age profile• Females

Group 2003 Survey (%) 2005 Survey (%)

Females 24 25

Education Less than Year 10

42 43

Looking for work 32 31

Semi-skilled 28 34

Unskilled 40 38

Single Living Alone

26 26

Aged 18 – 24 31 33

Aged 70 + 31 37

Renting 29 27

ANZ financial literacy surveys in 2003 and 2005: Groups over-represented in the lowest financial literacy quintiles


The involvement of community groups in facilitating delivery of the MoneyMinded program has assisted in targeting the over-represented groups

Community group involvement has also proven to be successful in the recruitment of participants for the Saver Plus matched savings program

Needs assessment through administration of a pre-training questionnaire, which captured demographic information and data on the areas that participants identified where they most needed financial literacy education in terms of the modules offered as part of the MoneyMinded program

Topic Area Number of Interested Participants

% of Total Participants

Planning and saving 101 71.1

Living with and managing your debt

79 55.6

Your consumer rights and responsibilities

68 47.9

Understanding financial paperwork

68 47.9

Your everyday banking and financial products

61 43.0

Your payment options 51 35.9

MoneyMinded modules identified by participants as being areas where they had an interest in acquiring greater financial knowledge


Together with the fact that the MoneyMinded workshop is composed of individual workshops the participation in which is chosen at the individual level suggests that the design performs well on the criteria relevant to the pre-implementation stage

Accountability Stage

Requires that details be collected on a range of matters including the education and services provided and base participant information

Basic participant information• Pre-training questionnaire collected data on

demographics, mathematical and computer literacy, Internet usage, current savings and spending behaviour, savings intentions and current usage of a range of financial products

• Pre-training questionnaire also captured data on the financial literacy of participants

Socio-demographic Characteristics of MoneyMinded Participants

Number of Participants % of Total

Gender

Male 19 13.4

Female 123 86.6

Age

15 – 34 34 24.3

35 – 44 63 45.0

45 – 54 29 20.7

55 + 14 10.0

Education

Primary / Some Secondary 30 22.8

All Secondary 24 18.2

TAFE / Workplace Training 32 24.2

University 46 32.3

Employment Status

Full time 50 35.4

Part time 45 31.9

Casual 10 7.1

Unemployed / Not looking 10 7.1

Student 1 07

Home duties 15 10.6

Retired 10 7.1

Accountability Stage (Cont’d)

The collection of such data provides the potential for a rich analysis to inform the program clarification, progress towards objectives and program outcome stages of any evaluation by allowing an exploration of the influences of these demographic and financial factors

Program Clarification Stage

Requires an assessment of the program strengths and weaknesses, and the use of such information in any re-assessment of program goals and objectives

This element is present in the MoneyMinded evaluation

Initial roll-out to a small number of community organisations and participants allows for the type of assessment and re-thinking that is at the core of the program clarification stage

Program Clarification Stage (Cont’d)

The MoneyMinded evaluation also involved the administration of post-training questionnaires to participants

• Required participants to provide an assessment of each workshop completed in terms of how useful they found the workshop to be, how satisfied they were with the topics and material covered in the workshop and an assessment of overall program satisfaction

Post-training evaluation data when coupled with the pre-training data provides scope for a rich analysis at the program clarification stage

High overall ratings for both satisfaction (90% reported being very satisfied or satisfied with the overall program) and usefulness (87% found the workshops to be extremely useful or useful)


A richer analysis for the program clarification is provided by considering whether there are differences in the satisfaction and usefulness ratings across categories in the demographic and financial data collected in the pre-training questionnaire

There were statistically significant differences in terms of usefulness and satisfaction ratings across participant categories

Higher ratings were provided by those in categories that captured lower incomes, education levels, employment status, mathematical and computer literacy, savings levels and intentions and base levels of financial literacy

Group χ2 d.f. χ2 value p-value

Program Satisfaction

Income Source 2 5.282 0.071

Employment Status 4 11.692 0.020

Income Level 6 12.823 0.046

Savings Intentions 6 12.634 0.049

Savings Level 6 4.756 0.576

Spending Behaviour 6 7.699 0.261

Understanding Bills 2 1.641 0.440

Credit card usage 2 5.700 0.058

Workshop usefulness



Income Level 12 51.544 0.000






Workshop Satisfaction



Income Level 12 22.480 0.032






Selection of the results on whether the ratings of satisfaction and usefulness provided by participants vary across the socio-demographic, financial product usage and financial literacy groupings


Appropriate statistical analysis can be beneficial at the program clarification stage of a structured evaluation process

• Bulk of data collected in the MoneyMinded evaluation was categorical in nature

• Statistical testing was conducted using the χ2 testing framework appropriate to such data

• Obvious issue in any formal statistical testing is whether the data collected satisfies the assumptions required of the statistical procedure being used

• Expected cell count of 5 for the asymptotic χ2 approximation to work

• Requirement not satisfied in the initial data• 2 approaches available: 1)aggregate cells together and then

conduct the tests 2) conduct a simulation analysis of the exact distribution of the test statistic typically involving some Monte Carlo technique


Post-training questionnaire also provided useful data on some dimensions of program delivery

• Website of resources made available to participants• 92% of participants did not access the website• Strong result which suggests use of technology in

delivery should be assessed, and not just assumed in the design features

• Also captured data on the number of individual workshops that participants attended; majority attended only a single workshop, with a small number attending 12 workshops


• Results show that participants who attended multiple workshops had higher usefulness and satisfaction ratings than those who attended a single workshop; differences in the ratings are statistically significant

• Suggests consideration be given in the program design to encouraging attendance at multiple workshops via greater bundling, while recognising that workshops are already bundled by broad topic area

• Useful and constructive information to obtain at the program clarification stage of an evaluation


Assessment of MoneyMinded also involved completion of questionnaires by facilitators who delivered the program, in terms of their expected usefulness of the workshops prior to delivery, and their perceptions of participant usefulness post-delivery

Ratings were high and were broadly consistent with the ratings provided by participants

Useful data at the program clarification stage in informing how the program goes forward


Final data collected of relevance to the program clarification stage was questionnaire data from both participants and facilitators in regard to improvements that could be made to the program in terms of expanding the topic coverage

Need for workshops on investment, superannuation and retirement planning

To be expected – ties in with government policy in recent years to move funding of retirement incomes to private funding via long-term superannuation investments

Progress Towards Objectives Stage

Objective of a financial literacy education program is in terms of improved financial literacy, and as a consequence better financial decision-making by participants

In the context of an initial evaluation of the roll-out of what is to become a wider program, such a stage is difficult to effectively action, except in terms of interpretation of ratings of satisfaction and usefulness that have been previously utilised at a program evaluation stage

Program Impact Stage

Short and long-term Looking for wider demonstrable evidence in terms of

improved financial literacy and better financial decision making by those who have participated in financial literacy programs

Fox, Bartholomae and Lee (2005) recommend that this measurement of impact be done relative to a control group who have not participated in such training

This stage is not relevant in the current assessment of MoneyMinded, given the program is only at an initial roll-out stage

Program Impact Stage (Cont’d) How such an evaluation might be built for use in the future

• Future broad surveys of financial literacy to include questions for the respondents on whether they have previously participated in a financial literacy education program

• Allows us to compare measures of financial literacy across groups that have/have not undertaken financial literacy education, after making appropriate adjustments for other socio-economic and demographic factors known to influence financial literacy

• Longer-term analysis of the savings behaviour of participants in a matched savings program would provide evidence of the joint impacts on financial decision-making of participation in both a matched savings program and financial literacy education

Conclusion

Exploration of the Fox, Bartholomae and Lee (2005) framework in the context of a specific program, the Australian MoneyMinded program

Framework developed in the context of US programs appears to generalise well to the Australian context, thus providing support to the general applicability of the framework

Formal statistical evaluation might be used at certain stages

Measurements of longer-term impacts i.e. behaviour change

evaluating a financial literacy program: the case of the australian moneyminded program rob brooks...

Documents

program participants

program impact slide

program clarification

framework conclusion

assessments slide

general financial literacy

evaluation process slide

financial literacy training