setting and assessing learning standards
DESCRIPTION
Presentation at meeting of Australian Learning and Teaching Fellows on 23 April 2012.TRANSCRIPT
Setting and assessing standards
Mark Freeman
[email protected] [email protected]
Status, options and challenges
Outline
1. Regulatory context
2. Setting learning standards
3. Assessing learning standards
4. Evaluating moderation options
5. Q & A
Support for this project has been provided by the Australian Business Deans Council, the Institute of Chartered Accountants in Australia, CPA Australia and the Australian Government Office for Learning and Teaching. The views expressed in this presentation do not necessarily reflect the views of any of these stakeholders.
I have taught Snoopy to whistle
I can’t hear him whistle
I said that I had taught him, not that he had learned
Definitions
Standards
“a definite level of excellence or attainment, or a definite degree of any quality viewed as a prescribed object of endeavour or as the recognised measure of what is adequate for some purpose, so established by authority, custom, or consensus” (Sadler, 2009)
Learning outcomes
“the expression of the set of knowledge, skills and the application of the knowledge and skills a person has acquired and is able to demonstrate as a result of learning” (AQF, 2011)
Learning standards
“the explicit levels of attainment required of and achieved by students and graduates, individually and collectively, in defined areas of knowledge and skills” (DEEWR, 2011)
It’s coming!
39%
TEQSA
Provider
Qualifications
Teaching and Learning
Research
Information
‘Fitness-for-purpose’ ‘standards’ (ie. internal external) Higher Education Standards Panel consult Ministers Commissioners Teaching standards separate from learning standards
Threshold
“At this point [the learning and teaching standards] are not threshold standards [but] what the government may choose to do in the future remains to be seen.“
The Australian 24 Aug 2011
TEQSA legislation
...take account of external standards.. e.g. published discipline standards...
...standards intended ...and ..actually achieved ....are benchmarked
...awards ...meet the corresponding specifications ...described in the AQF
TEQSA regulatory risk framework
T & L Standards Discussion Paper - principle 3 & 5
“TEQSA is not the only custodian of standards, nor are higher education institutions. This responsibility is distributed and shared more widely, including with disciplinary communities and professional associations”
“Institutional standards for teaching and learning will differ but all institutions must meet or surpass national standards”
Setting learning standards“Discipline communities will ‘own’ and take responsibility for implementing teaching and learning standards (working with professional bodies and other stakeholders where appropriate) within the academic traditions of collegiality, peer review, pre-eminence of disciplines and, importantly, academic autonomy” DEEWR (2009, p. 32)
9 discipline groups in 4 waves – 11 sets defined
Jul’09 Arts, Social Sciences & Humanities
Business, Management & Economics
Engineering & ICT
Jul’10 Architecture, Design & Building
Science
Feb’10 Creative & Performing Arts
Health, Medicine & Vet Science
Law
Feb’11 Education
Business, Management and Economics
Accounting Banking, Finance & Related Fields
Business Information Studies Business Management
Economics and Econometrics Hospitality Management
Human Resource Management
Industrial Relations
International Business Organisation Management
Marketing Sport and Recreation
Tourism Management Missing (eg. Logistics)
Degrees• Bachelor • Master (Entry) • Master (Advanced)
Cycles1. Agenda setting2. Awareness raising3. Consultation4. Dissemination
Engagement• 2,100 participants nationally • 38 Australian universities• 21 private/other providers• 20 others (eg professional
and peak bodies)
+
+
+
Judgement
Knowledge
Application
Communication & Teamwork
Self Management
Judgement
Knowledge
Application
Communication & Teamwork
Self Management
Judgement
Knowledge
Application
Communication & Teamwork
Self Management
+
+
+
Quantitative
Public sector
Professional Small business
Regional
Judgement
Knowledge
Application
Communication & Teamwork
Self Management
Chalk + talk learning
Online learning Problem-based learning
Team-based learning
Provider CProvider B Provider DProvider A
Indiv Small group Indiv
Q1 & 2
Assessing learning standards
Assessing learning standards
1. Perceptions – employers, graduates, professional bodies
“Assessment is largely dependent upon professional judgement and confidence in such judgement requires the establishment of appropriate forums for the development and sharing of standards within and between disciplinary and professional communities” (Tenet 6: Price et al, 2008)
“many graduates already subjected to skills testing for employment”
2. Common test – ACER, CLA, AHELO
3. External moderation – UK, Go8, Krause, ABDC-Prof Bodies
eg. AGS/CEQ; professional body accreditation
Moderation initiatives
QVS Krause-Scott et al Achievement Mat
Scope Multiple Multiple Accounting
Level Bachelor Bachelor Bach + Mast
HEI grouping Go8 11 across Start 10 across
Reviewers 1 academic 1 academic 2 aca/professionals
Data selection Stratified Stratified Randomised
Sample size 5% HD/D/C/P/F 1 HD/D/C/P/F per partner 5
Products All unit’s tasks All unit’s tasks Specific threshold
Intent Verification (QA) Qlty assurance & Qlty Enhancement (QE)
QA & QE
Authority Top-down Top-down Bottom-up
Achievement Matters Project
Aims
1. Evidence of accounting academic standards • External, double-blind, peer-reviewed
• Benchmark against national consensus (Bachelor & Master)
• All HEP types
2. A model process for obtaining and using evidence • Assessing inputs & outputs
• Quality enhancement & assurance
3. Professional learning and capacity building
Rationale: Improve, self-regulate, avoid perverse options
Pilot: Adelaide, Curtin, Deakin, Griffith, Monash, RMIT, Southern Cross, Sydney, USQ, UWA, UWS
Graduates of a Bachelor/Master (Entry) degree would be expected to justify and communicate accounting advice and ideas in straightforward/diverse collaborative contexts involving both accountants and non-accountants.
Pilot cycle: Threshold standard written communication
Master (entry): Diverse = Several competing or new qualitative perspectives and/or quantitative perspectives characterised by considerable data items, over multiple variables and knownrelationships between them.Bachelor: Straightforward = few qualitative perspectives and/or quantitative perspectives characterised by considerable data items over multiple variables and known relationships between them
23
Assess Enter Compare
Pre-F2F
F2F
Consensus Agree
Post-F2F
Implement
Reaching consensus on assessment task validity
24
Assess Enter Compare
Pre-F2F
F2F
Consensus Agree
Post-F2F
Apply • to assignment if student• to marking if faculty
Calibrating and grading to the standard
I’m confident rating assessment requirements and students’ work
Calibration – task validity
NA A
NA A
Individual results pre-workshop
• Min & max (n=26)
• Mean ±1 SD
Group results at workshop
• Small groups (n=5)
• Consensus
Calibration – UG student 1
Individual results pre-workshop
• Min & max (n=26)
• Mean ±1 SD
Group results at workshop
• Small groups (n=5)
• ConsensusNM M
NM M
Calibration – PG student 1
Individual results pre-workshop
• Min & max (n=26)
• Mean ± 1 SD
Group results at workshop
• Small groups (n=5)
• ConsensusNM M
NM M
Confirmation – PG student 5
Individual results at workshop
• Min & max (n=20)
• Mean ±1 SD
Group results at workshop
• Small groups (n=5)
• ConsensusNM M
NM M
Participant feedback
Having to enter my feedback into SPARK caused me to reflect on the reasons for my judgement
I expect this project will help establish national agreement on academic standards between accounting degree providers and with employers
Impact on academics
Strongly StronglyDisagree Agree
Pre-workshop 1
Pre-workshop 3
Post-workshop 3
33
Indiv Small group Indiv
Q3 & 4
ChallengesTEQSA
1. How will standards be set and monitored in a way that is sensible, fair, accepted and still economic?
HEI
2. How should we engage in disciplines setting standards?
3. How should agreed disciplinary learning standards be implemented into our curriculum?
4. How can we best participate in collaborative initiatives assessing achievement against national benchmarks?
5. How should evidence from our participation in national moderation projects be reported and used?
6. What systems changes and professional development are needed here to prepare for the standards agenda?
Thank you