Evaluating a Financial Literacy
Program: the Case of the
Australian MoneyMinded Program
Rob Brooks
Monash University
Presentation Structure Background / Context
Objectives
Overview of evaluation framework
Evaluation of the MoneyMinded program in the context of the framework
Conclusion
Background / Context
• The promotion of greater levels of financial inclusion is now seen to be a major public policy and corporate initiative across a variety of countries
• Core elements of programs to promote greater financial inclusion include matched savings programs and financial literacy education
Background / Context
• United States: American Dream Demonstration • United Kingdom: Saving Gateway• Australia: Saver Plus• Need for effective evaluation of programs in their
role in improving financial inclusion• Fox, Bartholomae and Lee (2005)• To date, no single framework to guide the
evaluation process
Background / Context
• Fox, Bartholomae and Lee extend the framework of Jacobs (1988) to develop a 5 tiered evaluation framework against which evaluations of financial literacy education programs can be conducted
Objectives• To explore this evaluation framework in the context of a
financial literacy program designed and developed in a different country, specifically the MoneyMinded financial literacy program, offered in Australia
• Extension of work of Fox, Bartholomae and Lee (2005)
• Provides information on the generalisability of framework
• Results will be of interest to a range of countries given comparable initiatives relating to financial inclusion across various countries
The Evaluation Framework
5 tiers, based on the work of Jacobs (1988)
1. Pre-implementation
2. Accountability
3. Program clarification
4. Progress towards objectives
5. Program Impact
Pre-implementation
Involves a needs assessment on the financial literacy training required by participants
• Needs identified from more general financial literacy testing
• Specific needs assessment around program participants
Accountability
Involves the collection of data on the education and services provided as part of the program, the costs involved and demographic information on participants
Program Clarification
Involves ongoing assessment of the program's strengths and weaknesses, and plans to review the program in the light of such assessments
Progress Towards Objectives
Involves the collection of data on the impacts of the program on participants
Program Impact
Builds on the previous stage in terms of long and short term impacts, typically involving comparison of participants and non-participants
The MoneyMinded Program General objective: To assist people to make informed
financial decisions and take control of their finances for their future
6 topics separated into 17 workshops*• Planning and Saving
• Easy Payments
• Understanding Paperwork
• Living with Debt
• Everyday Banking and Financial Products
• Rights and Responsibilities*(original version which has now been revised)
The MoneyMinded Program Allows individual participants to choose which workshops
they wish to complete Developed in part as a response to the results from the
first-ever major survey of consumer financial literacy in Australia (ANZ, 2003)
Results of the survey identified the following:• The groups within the community that could benefit
from improved financial literacy education • The issues that could be addressed in such an
education program Involvement of community groups in the delivery of
program
The MoneyMinded Program
ANZ plans to partner with around 100 community organisations during the next 5 years to deliver the program to 100,000 people nationally
Evaluation is critical for a program as comprehensive and extensive as MoneyMinded
Importance of evaluation being collected and analysed in a systematic manner
Evaluating the Money Minded Program
1. Pre-implementation needs assessment
• Not a formal needs analysis at the individual participant level
• Based on the results of the financial literacy survey reported in ANZ (2003)
• Telephone survey customised to particular needs and circumstances
Pre-implementation Stage (Cont’d)
ANZ survey identified the following groups were strongly over-represented in the lowest financial literacy quintile:
• Lowest education levels• Not working or in unskilled work• Lower income levels• Lower savings levels• Single• Extreme ends of the age profile• Females
Group 2003 Survey (%) 2005 Survey (%)
Females 24 25
Education Less than Year 10
42 43
Looking for work 32 31
Semi-skilled 28 34
Unskilled 40 38
Single Living Alone
26 26
Aged 18 – 24 31 33
Aged 70 + 31 37
Renting 29 27
ANZ financial literacy surveys in 2003 and 2005: Groups over-represented in the lowest financial literacy quintiles
Pre-implementation Stage (Cont’d)
The involvement of community groups in facilitating delivery of the MoneyMinded program has assisted in targeting the over-represented groups
Community group involvement has also proven to be successful in the recruitment of participants for the Saver Plus matched savings program
Needs assessment through administration of a pre-training questionnaire, which captured demographic information and data on the areas that participants identified where they most needed financial literacy education in terms of the modules offered as part of the MoneyMinded program
Topic Area Number of Interested Participants
% of Total Participants
Planning and saving 101 71.1
Living with and managing your debt
79 55.6
Your consumer rights and responsibilities
68 47.9
Understanding financial paperwork
68 47.9
Your everyday banking and financial products
61 43.0
Your payment options 51 35.9
MoneyMinded modules identified by participants as being areas where they had an interest in acquiring greater financial knowledge
Pre-implementation Stage (Cont’d)
Together with the fact that the MoneyMinded workshop is composed of individual workshops the participation in which is chosen at the individual level suggests that the design performs well on the criteria relevant to the pre-implementation stage
Accountability Stage
Requires that details be collected on a range of matters including the education and services provided and base participant information
Basic participant information• Pre-training questionnaire collected data on
demographics, mathematical and computer literacy, Internet usage, current savings and spending behaviour, savings intentions and current usage of a range of financial products
• Pre-training questionnaire also captured data on the financial literacy of participants
Socio-demographic Characteristics of MoneyMinded Participants
Number of Participants % of Total
Gender
Male 19 13.4
Female 123 86.6
Age
15 – 34 34 24.3
35 – 44 63 45.0
45 – 54 29 20.7
55 + 14 10.0
Education
Primary / Some Secondary 30 22.8
All Secondary 24 18.2
TAFE / Workplace Training 32 24.2
University 46 32.3
Employment Status
Full time 50 35.4
Part time 45 31.9
Casual 10 7.1
Unemployed / Not looking 10 7.1
Student 1 07
Home duties 15 10.6
Retired 10 7.1
Accountability Stage (Cont’d)
The collection of such data provides the potential for a rich analysis to inform the program clarification, progress towards objectives and program outcome stages of any evaluation by allowing an exploration of the influences of these demographic and financial factors
Program Clarification Stage
Requires an assessment of the program strengths and weaknesses, and the use of such information in any re-assessment of program goals and objectives
This element is present in the MoneyMinded evaluation
Initial roll-out to a small number of community organisations and participants allows for the type of assessment and re-thinking that is at the core of the program clarification stage
Program Clarification Stage (Cont’d)
The MoneyMinded evaluation also involved the administration of post-training questionnaires to participants
• Required participants to provide an assessment of each workshop completed in terms of how useful they found the workshop to be, how satisfied they were with the topics and material covered in the workshop and an assessment of overall program satisfaction
Post-training evaluation data when coupled with the pre-training data provides scope for a rich analysis at the program clarification stage
High overall ratings for both satisfaction (90% reported being very satisfied or satisfied with the overall program) and usefulness (87% found the workshops to be extremely useful or useful)
Program Clarification Stage (Cont’d)
A richer analysis for the program clarification is provided by considering whether there are differences in the satisfaction and usefulness ratings across categories in the demographic and financial data collected in the pre-training questionnaire
There were statistically significant differences in terms of usefulness and satisfaction ratings across participant categories
Higher ratings were provided by those in categories that captured lower incomes, education levels, employment status, mathematical and computer literacy, savings levels and intentions and base levels of financial literacy
Group χ2 d.f. χ2 value p-value
Program Satisfaction
Income Source 2 5.282 0.071
Employment Status 4 11.692 0.020
Income Level 6 12.823 0.046
Savings Intentions 6 12.634 0.049
Savings Level 6 4.756 0.576
Spending Behaviour 6 7.699 0.261
Understanding Bills 2 1.641 0.440
Credit card usage 2 5.700 0.058
Workshop usefulness
Income Source 2 14.345 0.001
Employment Status 4 29.169 0.000
Income Level 12 51.544 0.000
Savings Intentions 6 30.733 0.000
Savings Level 6 29.656 0.000
Spending Behaviour 6 74.844 0.000
Understanding Bills 6 48.953 0.000
Credit card usage 6 23.757 0.001
Workshop Satisfaction
Income Source 2 6.210 0.045
Employment Status 4 26.590 0.000
Income Level 12 22.480 0.032
Savings Intentions 6 30.453 0.000
Savings Level 6 29.337 0.000
Spending Behaviour 6 37.084 0.000
Understanding Bills 6 44.261 0.000
Credit card usage 6 32.603 0.000
Selection of the results on whether the ratings of satisfaction and usefulness provided by participants vary across the socio-demographic, financial product usage and financial literacy groupings
Program Clarification Stage (Cont’d)
Appropriate statistical analysis can be beneficial at the program clarification stage of a structured evaluation process
• Bulk of data collected in the MoneyMinded evaluation was categorical in nature
• Statistical testing was conducted using the χ2 testing framework appropriate to such data
• Obvious issue in any formal statistical testing is whether the data collected satisfies the assumptions required of the statistical procedure being used
• Expected cell count of 5 for the asymptotic χ2 approximation to work
• Requirement not satisfied in the initial data• 2 approaches available: 1)aggregate cells together and then
conduct the tests 2) conduct a simulation analysis of the exact distribution of the test statistic typically involving some Monte Carlo technique
Program Clarification Stage (Cont’d)
Post-training questionnaire also provided useful data on some dimensions of program delivery
• Website of resources made available to participants• 92% of participants did not access the website• Strong result which suggests use of technology in
delivery should be assessed, and not just assumed in the design features
• Also captured data on the number of individual workshops that participants attended; majority attended only a single workshop, with a small number attending 12 workshops
Program Clarification Stage (Cont’d)
• Results show that participants who attended multiple workshops had higher usefulness and satisfaction ratings than those who attended a single workshop; differences in the ratings are statistically significant
• Suggests consideration be given in the program design to encouraging attendance at multiple workshops via greater bundling, while recognising that workshops are already bundled by broad topic area
• Useful and constructive information to obtain at the program clarification stage of an evaluation
Program Clarification Stage (Cont’d)
Assessment of MoneyMinded also involved completion of questionnaires by facilitators who delivered the program, in terms of their expected usefulness of the workshops prior to delivery, and their perceptions of participant usefulness post-delivery
Ratings were high and were broadly consistent with the ratings provided by participants
Useful data at the program clarification stage in informing how the program goes forward
Program Clarification Stage (Cont’d)
Final data collected of relevance to the program clarification stage was questionnaire data from both participants and facilitators in regard to improvements that could be made to the program in terms of expanding the topic coverage
Need for workshops on investment, superannuation and retirement planning
To be expected – ties in with government policy in recent years to move funding of retirement incomes to private funding via long-term superannuation investments
Progress Towards Objectives Stage
Objective of a financial literacy education program is in terms of improved financial literacy, and as a consequence better financial decision-making by participants
In the context of an initial evaluation of the roll-out of what is to become a wider program, such a stage is difficult to effectively action, except in terms of interpretation of ratings of satisfaction and usefulness that have been previously utilised at a program evaluation stage
Program Impact Stage
Short and long-term Looking for wider demonstrable evidence in terms of
improved financial literacy and better financial decision making by those who have participated in financial literacy programs
Fox, Bartholomae and Lee (2005) recommend that this measurement of impact be done relative to a control group who have not participated in such training
This stage is not relevant in the current assessment of MoneyMinded, given the program is only at an initial roll-out stage
Program Impact Stage (Cont’d) How such an evaluation might be built for use in the future
• Future broad surveys of financial literacy to include questions for the respondents on whether they have previously participated in a financial literacy education program
• Allows us to compare measures of financial literacy across groups that have/have not undertaken financial literacy education, after making appropriate adjustments for other socio-economic and demographic factors known to influence financial literacy
• Longer-term analysis of the savings behaviour of participants in a matched savings program would provide evidence of the joint impacts on financial decision-making of participation in both a matched savings program and financial literacy education
Conclusion
Exploration of the Fox, Bartholomae and Lee (2005) framework in the context of a specific program, the Australian MoneyMinded program
Framework developed in the context of US programs appears to generalise well to the Australian context, thus providing support to the general applicability of the framework
Formal statistical evaluation might be used at certain stages
Measurements of longer-term impacts i.e. behaviour change