history on trial: evaluating learning outcomes through audit and … · 2017. 8. 17. · table 1....

89Teaching & Learning Inquiry, Volume 3, Issue 2, pp. 89–105, 2015. Copyright © 2015 The International Society for the Scholarship of Teaching and Learning

Sean Brawley, MACQUARIE UNIVERSITY, [email protected] Clark, UNIVERSITY OF NEW ENGLAND, [email protected]

Chris Dixon, UNIVERSITY OF QUEENSLAND, [email protected] Ford, UNIVERSITY OF NEW SOUTH WALES, [email protected]

Erik Nielsen, MACQUARIE UNIVERSITY, [email protected] Shawn Ross, MACQUARIE UNIVERSITY, [email protected]

Stuart Upton, UNIVERSITY OF NEW SOUTH WALES, [email protected]

History on Trial: Evaluating Learning Outcomes through Audit and Accreditation in a National Standards Environment

ABSTR AC T

This paper uses a trial audit of history programs undertaken in 2011 2012 to ex

plore issues surrounding the attainment of Threshold Learning Outcomes (TLOs)

in an emerging Australian national standards environment for the discipline of

history. The audit sought to ascertain whether an accreditation process managed

by the discipline under the auspices of the Australian His tori cal Association (AHA)

could be based on a limited intervention, “light touch” approach to assessing

attainment of the TLOs. The results of the audit show that successful proof of

TLO attainment would only be possible with more active intervention into exist

ing history majors and courses. Assessments across all levels of history teaching

would have to be designed, undertaken, and marked using a rubric matched to

the TLOs. It proved unrealistic to expect students to demonstrate acquisition of

the TLOs from existing teaching and assessment practices. The failure of the “light

touch” audit process indicates that demonstrating student attainment under a

national standards regime would require fundamental redevelopment of the

curriculum. With standards based approaches to teaching and learning emerg

ing as international phenomena, this case study resonates beyond Australia and

the discipline under investigation.

KEY WORDS

After Standards Project, history, learning outcomes assessment, quality assurance,

quality improvement, disciplinary standards, national standards

INTRODUC TION

In the last decade, higher education systems around the world have become increasingly involved in the development of quality assurance and quality improvement frameworks to evaluate what students actually gain from a university education. In Australia, engagement with standards based teaching at the tertiary level began with the recommen

90 TEACHING & LEARNING INQUIRY, VOL. 3.2 2015

Brawley, Clark, Dixon, Ford, Nielsen, Ross, Upton

dations of the 2008 Bradley Review (Bradley, Noonan, Nugent, & Scales, 2008, p. 136). Drawing on standards frameworks operating overseas and driven domestically by the 2003 introduction of the Australian Universities Quality Assurance Agency (AUQAA), Bradley advocated for a new sys tem for Australian tertiary education under which university programs would be taught to national standards with external evaluation as a dimension of quality assurance.

In 2010, history was chosen to participate in the Australian Learning and Teaching Council’s (ALTC) “Learning and Teaching Academic Standards Project” (LTAS) as a pilot discipline for the Arts, Humanities, and Social Sciences (Brawley et al., 2011, p. 177 188; Brawley et al., 2013a, p. 21 26). The project set out to explore how disciplines could respond to a national standards agenda by, in the first instance, designing what the ALTC called Threshold Learning Outcomes (TLOs). Within the Scholarship of Teaching and Learning (SoTL), “threshold concepts” are discipline based hurdles that students must pass in order to move from one state of discipline understanding to another (Land, 2011). According to Meyer and Land (2005, p. 373), such a shift in thinking is “transformative,” “irreversible,” and “integrative.” It is also likely that learning of such profound disciplinary significance may be “troublesome” (King & Felten, 2012, p. 5; Meyer & Land, 2005, p. 374). Indeed, Pace and Middendorf (2004) have observed that threshold concepts oft en significantly contribute to “bottlenecks” in learning. Once students have attained such discipline specific threshold knowledge and skills, however, they may be deemed to have crossed a significant boundary: “[T]hreshold concepts lead not only to transformed thought but to a transfiguration of identity” (Meyer & Land, 2005, p. 375). Such threshold concepts inform the construction of TLOs proposed by LTAS for courses or programs of study.

Some argue that the adoption of TLOs as the basis of national standards has great promise because TLOs explicitly link disciplinary knowledge to teaching and learning practice (Hill, 2012, p. 53). This innovation, however, has also attracted criticism. Hussey and Smith (2002, p. 221 223) argue that the new focus on TLOs hinders authentic learning and is symptomatic of an unwanted “managerialism,” in which the economic tail is wagging the educational dog. This misplaced focus produces a “monitored, audited and evaluated” educational process in which learning outcomes “are in danger of becoming little more than spurious devices to facilitate auditing at the expense of the educational process” (Hussey & Smith, 2002, p. 222). Hussey and Smith go on to distinguish between explicitly stating student expectations to facilitate learning, and using learning outcomes to monitor educational practices in order to make education “more manageable” (2002, p. 223). Elsewhere, they argue that learning outcomes do not “lend themselves to strict auditing” because they “cannot be, in themselves, either clear or precise and do not specify objectively measurable entities” (Hussey & Smith, 2005, p. 232). Furedi (2013) goes much further, describing learning outcomes as “corrosive” because they cultivate a “utilitarian ethos to academic life” and reduce the “open ended experience” of a university education (p. A2).

A broad based consultation process with history’s disciplinary community, led by Iain Hay for the ALTC, and coordinated through the discipline’s peak body, the Australian His tori cal Association (AHA), produced national TLOs for the bachelor’s level (“Australian Qualification Framework Level 7”). These TLOs were then endorsed by the

91

HISTORY ON TRIAL: EVALUATING LEARNING OUTCOMES

AHA on behalf of the discipline. Subsequent to this endorsement, the AHA revised the eight TLOs for the discipline of history, which are reproduced in Table 1 (Brawley et al., 2011, p. 171 188; Brawley et al., 2013a, p. 21 26).

The purpose of the LTAS process was to create the TLOs, but it made no plans for compliance, implementation, and proof of attainment by students. As a result, endorsement of the standards proved to be an aspirational exercise, divorced from the realities of teaching and learning practice and separated from the problems of deployment and assessment. It appeared that the TLOs constituted a genuine expression of what studying history entailed. As such, it was assumed that they were already embedded in our daily practice as academic historians and teachers, and that students would therefore attain the TLOs as a natural outcome of existing history teaching. Such an assumption proved to be optimistic—and problematic. Having participated in this national standards project, and having produced TLOs broadly acceptable to the discipline, history next faced the challenge of establishing audit procedures to demonstrate that students in the nation’s history majors had successfully met the new standards. In 2011 2012, the After Standards project, funded by the ALTC (re formed as the Office of Teaching and Learning [OLT] in 2011), designed and tested a “light touch” process to gage student achievement of the history TLOs (Brawley, 2013a; Brawley et al., 2013b). This process was designed to work with existing history curricula in Australian universities. It was also intended to support a low overhead, sustainable accreditation sys tem that could be administered with limited resources by the discipline itself, under the auspices of the AHA.

In this paper, we describe and assess that trial audit process as a case study in the challenges facing the imposition of teaching and learning standards in higher education everywhere. The trial audit revealed that although the TLOs were developed with the input and approval of the discipline, they proved aspirational in practice. The history TLOs did not reflect the realities of student learning in existing history majors. Without a thorough

Table 1. Threshold Learning Outcomes for History

Upon completion of a bachelor degree with a major in history, graduates will be able to

Knowledge

1. Demonstrate an understanding of at least one period or culture of the past. 2. Demonstrate an understanding of a variety of conceptual approaches to

interpreting the past. 3. Show how history and historians shape the present and the future. (Subsequently

revised by the AHA to “Show how history and historians shape their contemporary world.”)

Research4. Identify and interpret a wide variety of secondary and primary materials. 5. Examine historical issues by undertaking research according to the

methodological and ethical conventions of the discipline.

Analysis 6. Analyze historical evidence, scholarship, and changing representations of the past.

Communication 7. Construct an evidencebased argument or narrative in audio, digital, oral, visual, or written form.

Reflection 8. Identify and reflect critically on the knowledge and skills developed in their study of history.



going revision of the history curriculum to embed standards consistently, systematically, and progressively into assessment tasks, demonstrating TLO attainment at the end of the final year of an undergraduate degree was difficult, if not impossible.

Accordingly, we argue that if the Australian university sector intends to move to a standards based environment where students can prove that they have met their discipline specific TLOs upon graduation, then disciplines must undertake fundamental curriculum renewal, in clud ing significant training of—and buy in from—academic staff. The most feasible way to ensure student attainment of the TLOs, moreover, is to incorporate a marking rubric based on those TLOs into assessment tasks that have themselves been built with the standards in mind. Student attainment of national standards, and any accreditation process demonstrating such attainment, must go hand in hand with curriculum renewal, assessment design, and staff engagement.

THE A F T E R S TA N D A R D S PROJEC T

The After Standards project was led by historians from the University of New South Wales, the University of Queensland, and the University of New England, with support from the AHA and history academics in every relevant program in the country. The project’s aim was to formulate a response to the emerging TLO based standards environment. It would determine how TLOs should be embedded in teaching practice and audited, and provide the tools historians needed to engage and embrace the wide ranging implications of standards based teaching. The project organized a series of professional development workshops with plenary sessions and discussions for historians teaching across almost thirty Australian programs. The workshops also drew on the expertise of leading historians from Britain and the United States who had confronted standards environments or who were internationally renowned for their work in SoTL (Brawley et al., 2011, p. 191 193; Brawley et al., 2013a, p. 24 25).

It became clear at an After Standards project workshop held at the University of New South Wales in April 2011 that compliance, especially proof of student attainment, was a major practical problem under a standards regime. A working party was formed to explore how the disciplinary community could develop its own audit and compliance process. In an internal report submitted to the AHA in May 2011, the working party recommended that the discipline develop a credible, nation wide sys tem of self regulation. The report recommended that the AHA consider supporting an ex officio board member with expertise in teaching and learning, a proposal that was accepted, endorsed, and implemented. The report also highlighted the positive face of the standards environment; it encouraged historians to reflect on teaching and learning practices and offered a framework for curriculum renewal. The Commonwealth’s move to establish a new national regulator the, Tertiary Education Quality and Standards Agency (TEQSA), added a sense of urgency to the report. The most difficult question was how to organize a credible sys tem of self regulation under a national standards regime.

Although national, standards based accreditation is new to the discipline of history in Australia, the idea of accreditation for other disciplines and professions is not. Professions such as medicine, accounting, pharmacy, and engineering have long operated within the boundaries imposed by national accrediting bodies. Accreditation in these fields has been essential for students and universities, as well as for the international standing of the professions themselves. Moreover, internationally, much work has been done to con

93


sider the feasibility of assessing learning outcomes and approaches to do so. The OECD Assessment of Higher Education Learning Outcomes (AHELO) and the US Collegiate Learning Assessment (CLA) process are two notable examples (Coates & Richardson, 2012; Benjamin, 2013).

ACCREDITATION

The After Standards project developed and tested a self regulated accreditation system for the discipline (Brawley et al., 2013a, p. 26 32). The questions driving the exercise included the following:

1. Would it be possible for history, without experience of meeting external accreditation processes, to construct, support, and implement its own accreditation process?

2. Could history use audit and accreditation to drive its own quality assurance and improvement processes within the new standards environment?

3. Could history ultimately meet a hypothetical standards audit from a higher regulatory body and thereby stave off potential control from an external accreditor such as a state agency?

The three universities leading the project tested an accreditation sys tem that included two components: a compliance phase and an audit phase. These institutions also took part in the trial. All three have strong research intensive history departments and fine teaching reputations. The University of New England differs from the other two in that it has a large off campus, distance learning cohort and a larger number of non traditional, mature aged students. In the compliance phase, each university submitted a list of its courses, in clud ing stated learning outcomes and the assessment tasks aligned with them. The After Standards team then mapped that information against the TLOs. In most cases, this involved looking for a best, rather than a perfect, fit. A simple traffic light sys tem was utilized to align course based learning outcomes with specific TLOs. The course based learning outcomes were color coded green for a good match, orange for a possible match, and red in cases where a learning outcome could not be mapped to a TLO (Table 2; colors replaced by indicative column). The goal was to construct a light touch process, where standards based accreditation sat atop current practices and made use of the current learning outcomes.

The resulting tables were returned to the participating universities to confirm that the project team had properly mapped TLOs to course based learning outcomes. This involvement compelled institutions to consider the degree of alignment of their courses with the TLOs. Discipline conveners were thus able to track how the TLOs were represented across their own majors. Institutional course level learning outcomes appeared to address most of the TLOs, with some exceptions. Most notably, TLO 8, which involves students reflecting on their studies as historians, and TLO 3, which deals with historians’ relationship to the contemporary world, were oft en poorly represented (see, for example, Brawley et al., 2013b, p. 38, Figure 11). These findings reinforce opinions expressed by After Standards workshop participants that TLOs 3 and 8 were the most difficult to teach and assess.

With the exception of TLOs 3 and 8, the compliance trial confirmed that course level learning outcomes were broadly aligned with TLOs. As a result, it was thought that



programs were taught in such a way that students could demonstrate attainment of the TLOs, an optimistic conclusion that reflected a belief that the TLOs genuinely represented the teaching practice of historians in Australia. The broad compatibility between local outcomes and TLOs revealed by our compliance trial obscured the fact that the alignment was oft en approximate and that TLOs did not always map to course assessments and curriculum. In short, the trial suggested that it would not be difficult for history majors who attained local outcomes to “pass” an audit process based on the TLOs.

In the audit phase of the trial, each university submitted five final (third) year assignments from each of three nominated courses representing three randomly selected TLOs (chosen in light of the course learning outcome TLO map). Capstone courses were chosen by default when they existed. The in di vidual student assignments requested were the lowest passes from each course, an intentional decision driven by the pedagogical intention of TLOs (at the three participating institutions, a “pass” indicates a mark of 50% or higher in the course). Since TLOs represent minimum exit thresholds for all students who complete a major in history, we reasoned that the lowest pass students must be able to demonstrate that they had attained the TLOs. Assignments were de identified so that neither the originating university nor the in di vidual student was known to the audit team. Assignments were then distributed across the After Standards team and individually marked against the nominated TLO for that assignment. The student work was not marked according to whether it should have passed or failed the particular assignment or course—many factors determine whether any given assessment passes or fails in those contexts—but instead, whether the students demonstrated that they had attained the specific TLO under consideration.

To evaluate student attainment of TLOs, a four point scale (0 3) was used. This scale was informed by the “common format” (1 4 scale) used in the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics, which were developed by

Table 2. An Indicative Example of Unit/course Mapping against the Learning Outcomes and the TLOs

COURSE CO LO R CO D E LEARNING OUTCOME RELE VANT TLO

HIST3111 Medieval Europe

Green (good match) Confront the relationship between past and present

(3) Show how history and historians shape the present and the future

Green (good match) Analyze textual material(4) Identify and interpret a wide variety of secondary and primary materials

Green (good match) Review a significant period of European history

(1) Demonstrate an understanding of at least one period or culture of the past

Red (no match) Frame historical questions

Orange (possible match) Recognize critical thinking(6) Analyze historical evidence, scholarship, and changing representations of the past

Green (good match) Recognize logical argument and lucid writing

(7) Construct an evidencebased argument or narrative in audio, digital, oral, visual, or written form

95


the Association of Ameri can Colleges and Universities (AACU, 2013). The TLOs were used to create criteria for the marking scale. For example, TLO 3 (“Show how history and historians shape the present and the future”) was unpacked into four discrete levels of attainment (Table 3).

Each assignment was allocated a score depending on the extent to which the corresponding TLO was met. For purposes of the trial, the After Standards team determined that for any given assignment, an average score of two was necessary to pass (i.e., a cumulative score of 10 points out of a possible 15 across all five assignments from a single institution marked against a given TLO).

TRIAL RESULTS

When the scores were tallied for all pieces of student assessment across the selected TLOs, the results were surprising (Table 4). Scores were uniformly low. Of the 40 pieces of assessment audited, five scored zero, indicating that the work did not at all meet the TLO against which it was examined. Eleven pieces “passed” with a two or higher, indicating that the TLO was substantially attained. Only one assessment scored a three, indicating that the TLO was deemed to have been wholly met. Only one course—the sole capstone course represented in the trial—warranted a (bare) “pass” (10 out of 15) in one TLO, meaning that the audited TLO was achieved. Even this single instance of success involved what is likely the easiest of the TLOs to meet: TLO 1, which requires the student to demonstrate some his tori cal content knowledge.

TRIAL DISCUSSION

The results were singularly disappointing. If this exercise had been a genuine audit for accreditation, then no history program would have passed. We use the word “passed” advisedly because analyzing the data raised major questions, especially since we tested only a small sample of student assessments against the TLOs and used a sliding scale to make a judgment. What should constitute a “pass” for an assessment or a course in a standards environment? Were the results indicative of marking cultures that saw a bare pass as akin to a reward for effort? How many pieces of student work should be examined per course in order to determine that a major had “passed,” i.e., to demonstrate that all students graduating with the major had attained all of the TLOs? Should all of the assessments submitted be required to have met the TLO in question to rate a pass or, for example, only three out of five? Should a major be judged against all TLOs at once to

Table 3. Threshold Learning Outcome 3 Attainment Scale

SCORE CRITERIA

3 Shows a critical understanding of the impact of historical events and processes on current and future situations.

2 Shows understanding of the impact of historical events and processes on current and future situations.

1 Shows a limited understanding of the impact of historical events and processes on current and future situations.

0 Shows little or no understanding of the current and future relevance of historical events and processes.



rate a pass, or should it be required to measure up against only a selected few (as in the trial)? Should different TLOs be tested each year?

By the end of the trial, the After Standards team was convinced that if the discipline was to operate effectively within a national standards environment, then every audited student completing a major in history should have to pass every TLO in order to meet that standard. This approach was seen as necessary to endorse the major under audit against a comprehensive national standard. Further, all TLOs had to be audited at once rather than randomly assigned because not all TLOs are equal in difficulty. It is much easier, for example, to meet TLO 1 (“Demonstrate an understanding of at least one period or culture of the past”) than it is to meet TLO 3 (“Show how history and historians shape the present and the future”). If only a small number of TLOs were chosen for any given audit, it would be difficult to ensure that the selection of the TLOs over time was equitable for all majors participating in the accreditation process.

The poor scoring in the trial also raised questions about the usefulness of employing a sliding scale. Since the TLOs represent minima, we concluded that such a scale was not an appropriate measure of attainment. It does not matter how well a student performs against the TLO for the purposes of accreditation, only whether that student met the TLO at all. A binary scale thus represents a more appropriate way to test this kind of attainment; the student should simply meet or not meet a given TLO.

Since the TLOs are minima, furthermore, it was also inappropriate to allow some students’ exemplary achievement to counterbalance other students’ failure (e.g., threes on the sliding scale balancing out zeroes and ones to produce a “passing” average of two). For threshold standards to retain their meaning, all students had to meet all TLOs.

Table 4. TLO Attainment Trial Results

INS

TIT

UT

ION

CO

UR

SE

TLO

AS

SE

SS

ME

NT

S

RE

CE

IVE

D

EX

AM

INE

R

ASSESSMENT SCORES

AGAINST TLOs (THREE-

POINT SC ALE)

TO

TAL

(O

UT

OF

15

; 1

0 T

O P

AS

S)

PIE

CE

1

PIE

CE

2

PIE

CE

3

PIE

CE

4

PIE

CE

5

University (A)

X 3 Yes A 1 1 1 1 0 4

Y 6 Yes B 1 1 0 1 1 4

Z 5 No C — — — — — —

University (B)

A 8 Yes D 0 1 0 0 1 2

B 4 Yes E 0.5 1 1 1 1 4.5

C 2 Yes C 1 1 2 2 3 9

University (C)

M 1 Yes F 2 2 2 2 2 10

N 3 Yes A 2 2 2 2 1 9

O 7 Yes C 1 1 1 1 2 6

AVERAGE 6.1

97


Most importantly, the trial revealed problems interpreting the TLOs for the purposes of judging whether assignments demonstrated their attainment. The standards themselves were problematic, too soft edged for a metrics based audit process, and in need of refinement and clarification. In this regard, our study supports the argument of Hussey and Smith (2002) that Threshold Learning Outcomes are difficult to audit because of their broad nature. Problems with the wording of the standards sat around two major areas: ambiguous standards and compound standards.

Ambiguous wording obscured how markers should interpret what the standard was seeking; different markers could reach different conclusions about what the standard meant in the context of student work. The After Standards team decided that marking assignments would have been easier and more consistent if clear and unambiguous descriptors were provided for each standard.

Another problem with TLO wording was that some of the standards evaluated more than one learning outcome. In other words, many of the TLOs are, in practice, compound standards. Auditors faced a problem with students who met one part of a standard but not another; should such performance be deemed as having met the standard? Compound standards forced a decision: apply a flexible aggregate of their multiple components, or require separate judgments on each aspect of the standard. For example, TLO 4 asks students to “Identify and interpret a wide variety of sec ondary and primary materials.” Must a student both identify and interpret? What happens if the student does not interpret but only identifies? Similarly, what happens if only sec ondary sources are identified or interpreted, without any reference to primary sources? For TLO 4 alone, some 15 permutations are possible. These practical problems with the TLOs made them difficult to deploy as benchmarks for authentic student work. The ramifications of teaching and testing them had not been adequately explored when they were developed and approved by the discipline.

LESSONS FROM THE TRIAL

Beyond flaws in our initial audit procedure and practical difficulties with the TLOs themselves, the trial produced a number of useful pedagogical lessons about modifying current practice to comply with the standards:

1. Alignment. Mapping TLOs retrospectively to existing, local, course level learning outcomes is unlikely to be sufficient for audit purposes. Assessments that do not address the standards directly are unlikely to pass a compliance audit. Each major assessment task in a capstone or other final year course must be designed to assess the TLOs.

2. Curriculum development. Just as in di vidual TLOs could not be met by existing assignments, ensuring that students meet all TLOs by the time they graduate requires redesigning courses and majors. Even if the TLOs represent his tori cal teaching practice well, it is impossible to retrofit these standards onto currently existing courses and majors. Courses must be designed to address particular TLOs, and majors must be structured and coordinated so that all students practice all TLOs over the course of their undergraduate careers. A light touch compliance process is unrealistic. Serious curriculum renewal will be necessary to meet standards. In



a standards based environment, the standards must serve as the basis for the construction of the curriculum.

3. Teaching the standards. Historians need to teach students—explicitly and transparently—how to meet the eight standards.

4. Learning the standards. In order for student work to reflect attainment of the standards, the students themselves must know what the standards are and how to meet them. Transparency is again criti cal.

TOWARDS A MORE ROBUST AUDIT AND ACCREDITATION PROCESS

Taking onboard the lessons learned from the trial audit, the After Standards team organized a workshop in Darwin in No vem ber 2012, inviting eight exceptionally engaged historians from across Australia to formulate a new accreditation model (Brawley, et al., 2013b, p. 52 54). The principles agreed on were as follows:

1. All TLOs should be tested at once (each student’s work would be assessed against all eight TLOs).

2. TLOs should be tested on a binary “meets”/ “does not yet meet” scale. 3. All TLOs had to be successfully attained by all students. 4. Review for accreditation should be double blind (the reviewers would not know

the institutions, and the reviewers would remain anonymous). 5. Differences of professional judgment should be accommodated (e.g., for every

TLO, at least two of three reviewers must agree that the standard is met/not met). 6. A package of assessment needed to be developed for students to complete in the

final year of their undergraduate program that could serve as a vehicle for them to demonstrate all TLOs, but one which could also be audited expeditiously for compliance and accreditation.

7. Curriculum and assessment improvements had to be devised that would enable every student earning a major in history—even “low pass” students—to consistently and reliably demonstrate attainment of all TLOs.

Workshop participants considered three basic approaches to auditing and accreditation:

1. A final examination approach. 2. A multi course portfolio approach. 3. A capstone course output approach.

A final examination approach, while useful in other contexts, was ultimately discarded as an inferior instrument to measure student attainment of the TLOs specific to history. While tests like the Collegiate Learning Assessment (CLA) are sophisticated and useful (Klein, Benjamin, Shavelson, & Bolus, 2007; USDE, 2006), they also have limitations (Kuh, 2006). The TLOs require analy sis and understanding, proficient expression, and reflection. Any test likely to demonstrate all of the qualities embedded in the TLOs would be extensive and onerous for students, institutions, and reviewers. The After Standards team remained unconvinced that extensive, formal examinations strictly

99


for accreditation would elicit authentic student work. Instead, the team feared that a formal examination would be treated as an arbitrary burden (cf. “The VALUE Project Overview,” 2009; Rhodes, 2008).

Strong support emerged from the workshop for either an assessment package produced by students in a major capstone course or a portfolio of student work. The capstone model involved work from a single course, while the portfolio model required students to keep samples of work undertaken over a period of time and across multiple courses.

Delegates at the workshop recognized the advantages of the portfolio option. Portfolio based assessment has worked well in other disciplines (O’Sullivan et al., 2012; Rowley & Cunbar Hall, 2009). ePortfolios have also attracted considerable interest in Australia as a source of evidence for student learning and part of transition into employment (Hallam & Harper, 2008), and in the US they form the basis of the VALUE process (Rhodes, 2008). Use of portfolios to evaluate TLO attainment presupposes that the TLOs will be taught and developed across a range of courses. This sys tem would encourage progression but at the same time allow different courses to concentrate on delivering particular TLOs appropriate to their aims and content. Not all courses would need to cover all TLOs. Students could compile their portfolios from a range of work, encouraging diversity of assessment tasks and requiring students to engage explicitly with the TLOs. Certainly the traditional essay would be part of the package, but it might not be the only type of task undertaken by students. The portfolio approach, however, places the great est administrative time and resource demands on students, the institutions, and compliance review committees. It requires that processes be in place to collect and manage samples of student work across many courses. There are also logistical problems for students who change majors or universities.

Auditing capstone course outputs was seen by others as a less burdensome approach than the use of portfolios, as well as one more suited to the culture of the discipline than a cumulative exam. Capstones are well established in the United States and Britain, and are becoming more common in Australia (Hauhart & Granhe, 2010; van Acker, 2011). A capstone is a course taken in the last year of major study, ideally as the ultimate course in a major or program. Capstones bring together all of the learning that has taken place over the duration of the major, oft en culminating with a student led research project (e.g., Greenbank & Penketh, 2009). Learning outcomes or graduate attributes have been successfully embedded in capstones (van Acker, 2011), making them a likely venue for assessing the TLOs in our context. At the Darwin workshop, advocates of this approach argued that a major research project with a reflective exercise in the capstone course would prove an efficient means of assessment, while recognizing the potential for a wider variety of assessment. It was agreed by all that it would be possible to design an assessment package for a capstone course that would allow students to address all TLOs. Such a package would be easier to collect, distribute, and audit against the standards than a multi course portfolio. Capstones, however, present staffing challenges. Can we ensure that all students have a supervised research experience in the capstone when providing it is so labor intensive? Can we accommodate students’ desire to study particular periods or cultures? Although probably less administratively onerous than a portfolio, workshop participants recognized that the staffing resources required for a successful capstone may place it out of reach for some institutions.



The workshop resolved that any decision to institute either a capstone or a portfolio approach within an in di vidual major must be determined by the peculiar requirements of each university’s systems, structures, and student cohort. By the end of the workshop, delegates had decided to leave the specific assessment package compiled by students for TLO audits to in di vidual institutions, whether from a single course or a range of courses. Whatever package was submitted by an institution would, however, be marked against a standard rubric based on the TLOs. Rubrics have the advantage of standardizing marking of, for example, e portfolios (Tomkinson & Freeman, 2007). The VALUE project in the US has demonstrated that rubrics, as statements of expected learning, are effective at implementing shared core criteria across diverse institutions. They can also improve teaching and learning by providing students and instructors with clear, shared expectations, and are more likely than standardized tests to capture authentic student work (Burnett & Williams, 2009; Rhodes, 2008; Rhodes, 2009; “The VALUE Project Overview,” 2009;). While the VALUE project focused on the combination of rubrics with e portfolios, After Standards workshop participants believed that they could be equally applied to outputs from a single course, particularly a capstone (Levia & Quirring, 2008). An output agnostic, rubric based approach offers a number of advantages. Most importantly, it provides a uniform minimum standard to over 30 history programs in Australia, all of which have different identities, student cohorts, and institutional settings, while also recognizing the autonomy of each.

With the assistance of some preliminary work on TLO “descriptors” drafted by the origi nal LTAS project and further refined at the Darwin workshop, the After Standards team produced an expanded “National Rubric” that specified what was required to meet each TLO, in clud ing additional guidance to facilitate local implementation (Brawley, et al., 2013b, p. 93 94). Compound standards were also disaggregated, with the expectation that students must pass each component of the TLO in order to pass the TLO as a whole. This rubric was designed to ameliorate some of the practical problems encountered with the bare standards. It can be applied to a wide range of student outputs, from a single class or from courses across an entire major. The rubric unpacks the TLOs, helping academic staff to understand what must be taught and students to understand what must be learned.

A natural extension of the rubric involved suggesting “credit,” “distinction,” and “high distinction” levels of attainment (corresponding to the grades typically assigned in Australia). Although the After Standards team agreed that any attainment above a “pass” was not directly relevant during an audit, and that in di vidual institutions should be free to define their own criteria for higher levels of attainment, the production of model marking rubrics could encourage uptake and deployment. To initiate this process, the After Standards team also developed a template suggesting what such a local implementation might look like (Brawley et al., 2013b, p. 95). Considering the failure of the first trial audit, we considered such deployment crucial to ensure alignment between coursework and the TLOs, which is, in turn, required for students to achieve them and for majors to pass audits against them.

Finally, the After Standards team created a sample audit process built around the rubric and accommodating the variety of output discussed above (see Appendix A). Since all agreed that the TLOs genuinely reflect practice for Australian historians, in this case “teaching to the test” represents authentic learning and teaching, leading to outcomes endorsed by the discipline.

101


STATUS AND OUTLOOK

The implementation of a standards regime for history and other disciplines has been delayed in Australia (Brawley et al., 2013a, p. 21 24). Nevertheless, the community that has coalesced around the After Standards project has pressed ahead, in the belief that national standards are coming and convinced that the discipline must develop its own plan to face them. The accreditation model described above and in the appendices will now be considered by the AHA.

If the AHA is to supervise the accreditation of history majors, much preparatory work is needed. Historians across the country who have already endorsed the TLOs need to subscribe to the concept of an accreditation process. Curricula, courses, and assessments need to be redesigned with the TLOs in mind. Administrative systems must be put in place to select, collect, de identify, and process assessments for audit. Review panels must be recruited and organised. Provisions must be made to improve majors whose students do not meet the TLOs, and therefore fail the audits.

Whether the AHA and in di vidual stakeholders—institutions and majors—will embrace an accreditation model that requires broad and deep curriculum renewal remains to be seen, but if the discipline does not have an accreditation process ready to deploy when national standards are introduced, a far less desirable one may be imposed upon it.

CONCLUSION

A national standards regime is emerging fitfully for the history discipline in Australia. National standards offer an opportunity for the discipline to articulate the nature and value of what we teach, as well as to demonstrate student learning to our communities, institutions, and regulatory bodies. The After Standards project has spearheaded a proactive effort to devise discipline led accreditation while also improving the quality of history teaching and learning.

Initially, the After Standards team, in consultation with our stakeholder community, proposed a “light touch” approach to compliance auditing. A light touch proved insufficient, however. As a result, the After Standards project revisited potential audit approaches. A rubric has now been developed to guide educators through curriculum review and auditors through the assessment of student work for TLO attainment. This rubric can be applied to any reasonable package of student assessments, from a single course or from multiple courses, respecting institutional circumstances and autonomy while measuring compliance with national standards. The rubric, in expanded or modified form, can also be deployed for marking and evaluation of routine student work at in di vidual institutions in order to ensure alignment with the TLOs across assessments, courses, and curricula. Such alignment is crucial. Successful, measurable attainment of the TLOs will require it, along with additional curriculum renewal to ensure that all students majoring in history attain the TLOs during the course of their undergraduate careers. Since the discipline community has endorsed the TLOs as representative of authentic practice, their deployment offers significant opportunities for not just quality assurance and compliance, but also for quality improvement—more effective teaching and more profound learning of history in Australia.

Sean Brawley is Professor and Head of Department for the Department of Modern History, Politics,

and International Relations at Macquarie University (Australia).



Jennifer Clark is Professor of History in the School of Humanities at the University of New England

(Australia).

Chris Dixon is Associate Professor of History in the School of History, Philosophy, Religion, and Clas-

sics at the University of Queensland (Australia).

Lisa Ford is Associate Professor of History in the School of Humanities and Languages at the Uni-

versity of New South Wales (Australia).

Erik Nielsen is an Honorary Fellow at the Department of Modern History, Politics, and International

Relations at Macquarie University (Australia).

Shawn Ross is Associate Professor of History and Deputy Director of the Big History Institute at

Macquarie University (Australia).

Stuart Upton is Academic Programs Manager, Faculty of Engineering, at the University of New

South Wales (Australia).

REFERENCES

Association of Ameri can Colleges and Universities. (2013). VALUE: Valid assessment of learn-

ing in undergraduate education. AACU. Retrieved July 12, 2013 from http://goo.gl/Zoko0.

Benjamin, R. (2008). The contribution of the Collegiate Learning Assessment to teaching and

learning. Retrieved 26 June 2013 from http://goo.gl/9ivbK.

Bradley, D., Noonan, P., Nugent, H., & Scales, B. (2008). Review of Australian higher education:

Final report. Commonwealth of Australia. Retrieved 13 July 2013 from http://goo.gl/kKe4F.

Brawley, S., Clark, J., Dixon, C., Ford, L., Grolman, L., Ross, S., & Upton, S. (2011). Applying

standards to tertiary level history: Policy, challenges and the After Standards project.

History Australia, 8(3), 177 194.

Brawley, S., Clark, J., Ross, S., Ford, L., & Dixon, C. (2013a). Learning outcomes assessment and

history: TEQSA, the After Standards Project and the QA/QI challenge in Australia. Arts and

Humanities in Higher Education, 12(1), 20 35.

Brawley, S., Clark, J., Dixon, C., Ford, L., Nielsen, E., Ross, S., & Upton, S. (2013b). After Stan

dards: Engaging and embedding history’s standards using international best practice to

inform curriculum renewal. ALTC (now OLT) Final Project Report, 2011 12. Retrieved 15

No vem ber 2014 from http://goo.gl/nlFRVz.

Burnett, M. N., & Williams, J. M. (2009). Institutional uses of rubrics and e portfolios: Spel

man College and Rose Hulman Institute. Peer Review, 11(1), 24 27.

Coates, H., & Richardson, S. (2012). An international assessment of bachelor degree gradu

ates’ learning outcomes. Higher Education Management and Policy, 23(3), 1 19.

Furedi, F. (2013) Learning outcomes are corrosive. CAUT Bulletin,60(1), A2.

Greenbank, P., & Penketh, C. (2009). Student autonomy and reflections on researching and

writing the undergraduate dissertation. Journal of Further and Higher Education, 33(4),

463 472.

Hallam, G. C., & Harper, W. E. (2008). Picturing the future: Exploring the opportunities to de

velop en ePortfolio community of practice in Australasia. In Proceedings: Engaging com-

103


munities, Conference of The Higher Education Research and Development Society of Austral-

asia (HERDSA), Rotorua, New Zealand. Retrieved 13 July 2012 from http://goo.gl/3DSh2.

Hauhart, R. C., & Grahe, J. E. (2010). The undergraduate capstone course in the social sciences:

Results from a regional survey. Teaching Sociology, 38(1), 4 17.

Hill, D. C. (2012). Learning outcomes: Perceptions about the influence of ABET accreditation

on OSH education. Professional Safety, 57, 53 61.

Hussey, T., & Smith, P. (2002). The trouble with learning outcomes. Active Learning in

Higher Education, 3, 220 233.

King, C., & Felten, P. (2012). Threshold concepts in educational development: An

introduction. Journal of Faculty Development, 26(3), 5 7.

Klein, S., Benjamin, R., Shavelson, R., & Bolus, R. (2007). The Collegiate Learning Assessment:

Facts and fantasies. Evaluation Review, 31, 415 439.

Kuh, G. (2006). Director’s message Engagement: The bridge from here to there. In Engaged

learning: Fostering success for all students. National Survey of Student Engagement. Annual

Report 2006 (pp.7 10), Bloomington, IN.

Land, R. (2011). There could be trouble ahead: Using threshold concepts as a tool of analy

sis. International Journal for Academic Development, 16, 175 178.

Levia, D. F., & Quiring, S. M. (2008). Assessment of student learning in a hybrid PBL capstone

seminar. Journal of Geography in Higher Education, 32(2), 217 231.

Meyer, J. H. F., & Land, R. (2005). Threshold concepts and troublesome knowledge (2):

Epistemological considerations and a conceptual framework for teaching and learning.

Higher Education, 49, 373 388.

O’Sullivan, A. J., Harris, P., Hughes, C. S., Toohey, S. M., Balasooriya, C., Velan, G., Kumar, R., &

McNeil, H. P. (2012). Linking assessment to undergraduate student capabilities through

portfolio examination. Assessment & Evaluation in Higher Education, 37(2), 379 391.

Pace, D. & Middendorf, J. (2004). Decoding the disciplines: A model for helping students learn

disciplinary ways of thinking. New Directions for Teaching and Learning, 98, 1 12. Rhodes,

T. L. (2008). VALUE: Valid assessment of learning in undergraduate education. New Direc-

tions for Institutional Research, S1, 59 70. doi:10.1002/ir.262.

Rhodes, T. L. (2009). From the Director. Peer Review, 11(1), 3.

Rowley, J. L., & Dunbar Hall, P. (2009). Integrating e portfolios: Putting the pedagogy in its

place. In R. J. Atkinson & C. McBeath (Eds.), Same places, different spaces. Proceedings asci-

lite Auckland 2009 (pp. 898 901). The University of Auckland, Auckland University of Tech

nology, and Australasian Society for Computers in Learning in Tertiary Education (ascilite).

Retrieved July 12, 2013 from http://www.ascilite.org.au/conferences/auckland09/procs/.

Tomkinson, B., & Freeman, J. (2007). Using portfolios for assessment: Problems of reliability

or standardisation? In Enhancing Higher Education, Theory and Scholarship, Proceedings of

the 30th HERDSA Annual Conference, Adelaide, 8- 11 July 2007 (pp. 574 585). Higher Educa

tion Research and Development Society of Australasia, Inc.

U. S. Department of Education. (2006). A test of leadership: Charting the future of U.S. higher

education. Wash ing ton, DC. Retrieved 13 July 2013 from http://goo.gl/Gmnn0.



The VALUE Project Overview. (2009). Peer Review, 11(1), 4 7.

van Acker, L. (2011). Embedding graduate skills in capstone courses. Asian Social Science,

7(4), 69 76.

APPENDIX A: MODEL ACCREDITATION PROCESS

Principles • Accreditation will be based upon an assessment package (e.g., a research essay plus associated reflective work from a capstone, or a student portfolio).

• If work is collected from a single course, the course selected should be third year, preferably a capstone, preferably offered in the final semester of the student’s career.

• If work is collected as a portfolio, it may come from any combination of courses, but should not exceed XXXX words in length (maximum length TBD).

• Each assessment package submitted for review will be tested against all eight TLOs. • Each TLO will be evaluated on a binary “meets/does not yet meet” scale, using descriptors encapsulated in a rubric.

• The rubric should be made publicly available so that student assessments / course learning outcomes / major learning outcomes / etc., can be aligned with it. It is recommended that student assessments be marked using a customized version of the supplied full marking rubric.

• Review for accreditation is double blind—the reviewers do not know the institutions, and the reviews will be anonymous.

• Differences of professional judgment will be accommodated. For every TLO, at least two of three reviewers must always agree that the standard is met.

Preparation 1. Institutions should avail themselves of the publicly available rubrics (“meets/does

not meet” and full marking rubrics). 2. Institutions must be prepared to provide a package of assessment from students

(maximum length XXXX words). This package may, for example, consist of a portfolio of student work or the outputs from a capstone course.

3. Institutions must provide sufficient administrative support—clean, anonymized submission of the five lowest pass assessment packages within a specified time frame. Students should also be informed that their assessments may be reviewed for accreditation purposes (anonymously).

a. If the institution submits a single assessment, “lowest pass” means the lowest pass on the task.

b. If the institution submits a package of multiple assessments from a single course, lowest pass means the lowest passes for the course.

c. If the intuition submits a student portfolio drawing from multiple courses, lowest pass means the lowest GPA or equivalent amongst all students graduating in the major in the year audited.

4. The process will be conducted online—a Google Apps model is presented below.

105


a. The accrediting body (e.g., the AHA) opens a Google Apps account (free or low cost) so that we can have a “branded” Google Drive, email addresses, etc.

b. Reviewers should be given email addresses associated with this account (for security reasons, can be forwarded).

c. A Google Doc of the TLO rubric with full descriptors will be posted on the Google Docs account for reviewers’ information.

d. A Google Survey based on a brief version rubric (without descriptors) will be used to collect and tabulate results.

e. Output will be as spreadsheets (full results and summary).Process 1. Institutions upload clean, anonymous copies of the five lowest pass assessment

packages to a secure, online folder. 2. Reviewers are given a link to two folders: one with the rubric, one with the sub

mitted assessments. 3. Reviewers prepare by examining the “elaborated” rubric with all of the descriptors

and “meets/does not yet meet” criteria. 4. Reviewers read each assessment and score it. Assessments and institutions will be

identified only by number for tracking. 5. A report will be generated with each institution given a top line “meets”/”does

not yet meet” grade, plus the details of how each reviewer marked each TLO (anonymous).

history on trial: evaluating learning outcomes through audit and … · 2017. 8. 17. · table 1....

Documents