teaching practice evaluation framework · web viewa consultative approach was undertaken to develop...

—Centre for Program EvaluationMelbourne Graduate School of Education

30 September 2019

Teaching Practice Evaluation Framework | Centre for Program Evaluation

Teaching Practice Evaluation FrameworkReport prepared for the Australian Government Department of Education and Training

AcknowledgementsWe would like to thank Sarah Ryan and Richard Bolto from the Department of Education and Training for their ongoing input and support for this study. Staff from the Department of Education and Training who participated in interpretation workshops are also acknowledged. Clinton Milroy, Christian Meyer, Erin Corbyn and Anisha Ghani from the Australian Institute of Teaching and School Leadership are also acknowledged for their invaluable contributions to this study. Most importantly, we wish to thank all the educators and education stakeholders, both in Australia and internationally, who have contributed to this study.

We acknowledge the following Research Assistants from the Centre for Program Evaluation who have supported the drafting of this report:

Esther KeeHanchao HouKatla HowerMeg StonnillMia ChenMichelle HsienNarelle RobertsonOskar WeimarRachel HeardJesse TseAllison Clarke

Version Number Date Submitted Status

1 30th September 2019 Review by the Department of Education and Training

2 4th December 2019

Recommended citation:

Clinton, J., Aston, R., Qing, E. & Keamy, K. (2019). Teaching Practice Evaluation Framework: Final Report. Report prepared for the Australian Government Department of Education and Training. Centre for Program Evaluation, Melbourne Graduate School of Education, The University of Melbourne.

ISBN 978-1-76051-900-1 Teaching Practice Evaluation Framework (PDF)

ISBN 978-1-76051-901-8 Teaching Practice Evaluation Framework (DOCx)

Contact DetailsProfessor Janet Clinton, Director Centre for Program Evaluation, Melbourne Graduate School of Education The University of Melbourne Parkville 3010 VIC

Phone: +61 3 9035 3697 Email: [email protected]

Teaching Practice Evaluation Framework | Centre for Program Evaluation i

Table of ContentsAcknowledgements.............................................................................................................i

Part A: The Evidence for Promoting Teachers’ Professional Growth...................................1

2. Introduction..............................................................................................................12.1. Background.....................................................................................................................1

3. Methodology.............................................................................................................73.1. Literature Scans.............................................................................................................. 8

3.2. Stakeholder Consultations............................................................................................10

3.3. Data Triangulation Process...........................................................................................15

4. Results....................................................................................................................164.1. Targeted Literature Scans.............................................................................................16

4.2. Stakeholder Interviews.................................................................................................28

4.3. Peak Body and Group Consultations.............................................................................31

4.4. Stakeholder Workshops................................................................................................33

4.5. Delphi Survey................................................................................................................39

5. Triangulation and Synthesis.....................................................................................52Preconditions for Evaluation Systems....................................................................................52

Principles for Selecting Evaluation Resources........................................................................52

6. Conclusions.............................................................................................................60

7. Final Word...............................................................................................................64

Part B: Framework for Promoting Teachers’ Professional Growth....................................677.1. Development of the Evaluation Process Model............................................................67

7.2. Implementing the Evaluation Process Model...............................................................69

7.3. Focussing.......................................................................................................................69

7.4. Initiating........................................................................................................................71

7.5. Evaluating..................................................................................................................... 73

7.6. Supporting Learning......................................................................................................75

7.7. Rubric for Appraising Evaluation Tools and Resources.................................................77

8. Closing Comments...................................................................................................83

9. References..............................................................................................................84

10. Appendices..............................................................................................................90Appendix A: Literature Scan Criteria.................................................................................90

Teaching Practice Evaluation Framework | Centre for Program Evaluation ii

Appendix B: Organisations Represented at Workshops....................................................91

Appendix C: Delphi Survey Data – Ranked Importance of Principles................................93

Appendix D: Teacher Evaluation Tools in Australian Contexts........................................108

Appendix E: Prompts for Principles.................................................................................110

Teaching Practice Evaluation Framework | Centre for Program Evaluation iii

List of TablesTable 1. Implementation of the Australian Teacher Performance and Development

Framework Across the System........................................................................................2

Table 2. Dimension Definitions in Explanatory Model of Teacher Characteristics...................5

Table 3. Literature Scan Search Criteria...................................................................................8

Table 4. Relevant Stakeholder Groups and Perspectives.......................................................12

Table 5. Sources Reviewed by Type and Sector......................................................................16

Table 6. Key Features of Evaluation Systems that Promote Teachers’ Professional Growth. 19

Table 7. Preconditions for Evaluation Systems that Promote Professional Growth...............20

Table 8. Principles for Teacher Assessment and Evaluation Derived from the Literature Scan......................................................................................................................................27

Table 9. Individual and Small Group Stakeholder Consultations............................................28

Table 10. Themes from Individual and Small Group Interviews.............................................29

Table 11. Stakeholder Workshop Activities Overview............................................................33

Table 12. Definitions of Teacher Effectiveness and Professional Growth...............................34

Table 13. Perspectives from Teacher Professional Growth Vignettes....................................35

Table 14. Tool Selection Process Themes...............................................................................37

Table 15. Legend for Tool Selection Process Themes.............................................................38

Table 16. Workshop Participants' Considerations for Selecting Tools....................................38

Table 17. Roles of Delphi Survey Participants (Round 1 and 2)..............................................40

Table 18. Organisations of Delphi Survey Participants (Round 1 and 2).................................40

Table 19. Primary Geographic Location of Delphi Survey Participants (Round 1 and 2)........41

Table 20. Percentages of Responses for Each Item Proposed for Removal............................42

Table 21. Mean Scores for Principles Weighted Across Survey Rounds.................................43

Table 22. Intra-class Correlations for Each Principle and Overall (Round 1)...........................44

Table 23. Intra-class Correlations for Each Principle (Round 2)..............................................45

Table 24. Summary Responses to Open-Ended Delphi Survey Questions..............................46

Table 25. Dimension Groupings of Principles.........................................................................50

Table 26. Triangulation Matrix for Preconditions...................................................................52

Table 27. Triangulation Matrix for Principles for Guiding Assessment, Feedback and Evaluation Practices......................................................................................................53

Table 28. Evaluation Process Scenario...................................................................................69

Table 29. Rubric for Appraising Evaluation Tools and Resources...........................................79

Table 30. Organisations Represented at the Sydney Workshop (20th June 2019)..................91

Teaching Practice Evaluation Framework | Centre for Program Evaluation iv

Table 31. Organisations Represented at the Darwin Workshop (27th June 2019)..................91

Table 32. Organisations Represented at the Melbourne Workshop (18th July 2019).............91

Table 33. Organisations Represented at the Perth Workshop (23rd July 2019)......................92

Table 34. Mean Scores for Items in the Evidence-Based Principle (Round 1 and 2)...............93

Table 35. Mean Scores for Items in the Authentic Principle (Round 1 and 2)........................94

Table 36. Mean Scores for Items in the Relevant Principle (Round 1 and 2)..........................95

Table 37. Mean Scores for Items in the Cost-Effective Principle (Round 1 and 2)..................96

Table 38. Mean Score for Items in the Professional Growth Principle (Round 1 and 2 )........97

Table 39. Mean Scores for Items in the Regular Principle (Round 1 and 2)............................98

Table 40. Mean Scores for Stems in the Context-Specific Principle (Round 1 and 2).............98

Table 41. Mean Scores for Items in the Comprehensive Principle (Round 1 and 2)...............99

Table 42. Mean Scores for Items in the Student-Based Principle (Round 1 and 2)..............100

Table 43. Mean Scores for Items in the Collaborative Principle (Round 1 and 2).................101

Table 44. Mean Scores for Items in the Understandability Principle (Round 1 and 2).........103

Table 45. Mean and Standard Deviation for All Items..........................................................104

Table 46. Evaluation Tools and Methods (in collaboration with AITSL) from the Investigating the Key Characteristics of Effective Teachers Review.................................................108

List of FiguresFigure 1. The Evaluation Process Model................................................................................xiv

Figure 2. Success criteria for the implementation of the Evaluation Process Model.............xvi

Figure 3. Teacher and teaching factors that relate to educational impact...............................4

Figure 4. Explanatory model of effective teacher characteristics (Clinton, et al., 2018)..........5

Figure 5. Design of the study....................................................................................................7

Figure 6. Phases of the project.................................................................................................8

Figure 7. Literature scan data extraction and analysis process..............................................10

Figure 8. The 14 key areas of a robust assessment program (ACER, 2017)............................26

Figure 9. Stakeholder group consultations ranked most important principles......................32

Figure 10. Stakeholder group consultations ranked least important principles.....................32

Figure 11. Scattergram of the principles at the item level......................................................48

Figure 12. 3-Cluster solution for the principle at the item level.............................................49

Figure 13. Scattergram for the principles...............................................................................49

Figure 14. 3-Cluster solution for the principles with dimension groupings............................50

Figure 15. Evaluation Process Model......................................................................................59

Teaching Practice Evaluation Framework | Centre for Program Evaluation v

Figure 16. Success criteria for the implementation of the Evaluation Process Model...........61

Figure 17. Information flow and feedback cycle in the Education System.............................63

Figure 18. A model of Collective Impact, taken from Kania and Kramer, 2011......................64

Figure 19. Teacher and teaching factors that relate to educational impact...........................65

Figure 20. Explanatory model of effective teacher characteristics.........................................66

Figure 21. Evaluation Process Model.....................................................................................68

Figure 22. Percentage of responses by item in the evidence-based principle (Round 1).......93

Figure 23. Percentage of responses by item in the evidence-based principle (Round 2).......94

Figure 24. Percentage of responses by item in the authentic principle (Round 1).................94

Figure 25. Percentage of responses by item in the authentic principle (Round 2).................95

Figure 26. Percentage of responses by item in the relevant principle (Round 1)...................95

Figure 27. Percentage of responses by item in the relevant principle (Round 2)...................96

Figure 28. Percentage of responses by item in the cost-effective principle (Round 1)..........96

Figure 29. Percentage of responses by item in the professional growth principle (Round 1).97

Figure 30. Percentage of responses by item in the professional growth principle (Round 2).97

Figure 31. Percentage of responses by item in the regular principle (Round 1).....................98

Figure 32. Percentage of responses by item in the context-specific principle (Round 1).......99

Figure 33. Percentage of responses by item in the context-specific principle (Round 2).......99

Figure 34. Percentage of responses by item in the comprehensive principle (Round 1)......100

Figure 35. Percentage of responses by item in the comprehensive principle (Round 2)......100

Figure 36. Percentage of responses by item in the student-based principle (Round 1).......101

Figure 37. Percentage of responses by item in the student-based principle (Round 2).......101

Figure 38. Percentage of responses by item in the collaborative principle (Round 1).........102

Figure 39. Percentage of responses by item in the collaborative principle (Round 2).........102

Figure 40. Percentage of responses by item in the understandability principle (Round 1)..103

Figure 41. Percentage of responses by item in the understandability principle (Round 2)..103

Teaching Practice Evaluation Framework | Centre for Program Evaluation vi

List of

AbbreviationsAbbreviation Description

AAAE Australian Alliance of Associations in Education

AITSL Australian Institute for Teaching and School Leadership

APST Australian Professional Standards for Teachers

ATPDF Australian Teacher Performance and Development Framework

CESE Centre for Education Statistics and Evaluation

CPE Centre for Program Evaluation

DET Australian Government Department of Education and Training

ECB Evaluation Capacity Building

HALT Highly Accomplished and Lead Teacher

OECD Organisation for Economic Cooperation and Development

SLTESC School Leadership and Teaching Expert Standing Committee

IntroductionIn November 2018, the Australian Government Department of Education and Training (DET) commissioned the Centre for Program Evaluation (CPE) to develop a framework to support the implementation of a teacher evaluation system to promote professional growth and select appropriate resources to support teacher evaluation in Australian schools (hereafter referred to as ‘the Framework’).

This study built on the findings of two previous reviews commissioned by the DET and conducted by CPE: Teacher Effectiveness Systems, Frameworks and Measures: A Review (Clinton et al., 2016) and Investigating the Key Characteristics of Effective Teachers: A Systematic Review (Clinton et al., 2018). The previous studies identified what effective systems and processes for teacher evaluation look like, along with describing and defining those variables that comprise the nexus between teachers, effective teaching practices and student learning outcomes that could be evaluated and assessed. As the third piece of this tripartite body of work, this study has built upon this foundational knowledge base to develop a practical Framework for use by educators and policy makers. The Framework will guide users in selecting and using robust evaluation tools and implementing effective evaluation practices, both of which facilitate teacher practice improvement and continued professional growth. As underpinned by a multi-sectoral, consultative approach undertaken for this study, the Framework design is reflective of the voice and needs of the education profession. This report comprises two sections: a report detailing the methodology and overall findings of the consultations and evidence base, and the Framework for conducting teacher evaluation, accompanied by recommendations for implementation.

Background

Teaching Practice Evaluation Framework | Centre for Program Evaluation vii

Executive SummaryDecember 2019

There is growing evidence that effective evaluation of teachers can enable continuous, targeted professional development while allowing teachers to reflect on relevant evidence that can be used to develop or adapt their practice. Such evidence has grown to the extent that evaluative practices by educators are considered essential for school improvement (Hattie, 2015; Earl & Timperley, 2015). In Australia in the last decade, supporting teacher effectiveness has become a cornerstone of a number of large-scale education reforms, such as the Australian Professional Standards for Teachers (APST; Australian Institute for Teaching and School Leadership [AITSL], 2018), which guide teachers’ practice in and outside the classroom, and the Australian Teacher Performance and Development Framework (ATPDF; AITSL, 2018), which provides a structure for educators to appraise, develop, and improve practice in an ongoing cycle. The previous reviews (Clinton et al., 2016 & Clinton et al., 2018) have highlighted that there is a gap between policy and practice in teacher evaluation for the purposes of ongoing professional development (Clinton & Dawson, 2018; Freeman, O’Malley, & Eveleigh, 2014). In addition, there is a lack of clarity regarding the selection of teacher evaluation tools for specific assessment needs to enact improvement in teaching.

Teaching Practice Evaluation Framework | Centre for Program Evaluation viii

Aims and ObjectivesThis study brought together the learnings gained from the previous reviews (Teacher Effectiveness Systems, Frameworks and Measures ([Clinton et al., 2016], and Investigating the Key Characteristics of Effective Teachers [Clinton et al., 2018]) with the voice of the education profession and other relevant sectors. This was achieved through the adoption of a research design with a series of consultation activities with key stakeholders embedded in the design. The key findings from this study have informed the development of a Framework that, at both an individual and system level, is intended to guide educators to engage in teacher evaluation practices that facilitate professional growth and development.

The overarching aim of the study was to support the development of a culture of evaluative thinking within teachers, leaders, schools and systems.

MethodologyThis study was underpinned by a multi-phase mixed methods research design, in which both qualitative and quantitative data were collected sequentially and simultaneously through several stakeholder consultation methods, along with rapid synthesis of relevant literature. Sectors outside education (police, nursing, medical education and allied health) were examined to gain a comprehensive understanding of professional development assessments and frameworks that focus on supporting professional growth.

Key activities were iteratively completed between February–September 2019:

Findings from all data collection activities were triangulated (through matrices) to determine the degree to which the findings were consistent across data sources. Consistency was examined through the degree to which the findings were similar across sources, and by assessing the strength of evidence for each key finding across the data sources. Overall, there was convergence in findings across data sources for most pre-conditions and for the identified principles for tool and resource selection.

Results & Key FindingsArising from these data collection activities was the need for the Framework to contain components to guide users to not only select appropriate evaluation tools, but to engage in wider evaluation practices. These evaluation practices include planning, implementation, and synthesis processes that ensure an ongoing cycle of appraisal, information use and supporting professional growth. Two outputs arose from this study:

(1) preconditions for evaluation systems

Teaching Practice Evaluation Framework | Centre for Program Evaluation ix

The objectives of this study included:

• To develop a process for the implementation of an evaluative system, and

• To understand the conditions necessary to implement the system, including the appropriate use of tools and resources.

The Framework is not intended to be used in performance management processes or as an all-encompassing and inflexible prescription of evaluative practice.

Ensuring utility and relevance

through dissemination &

reporting

Data triangulation and building

the Framework

Consultations with key

stakeholders: interviews and

workshops

Delphi survey (two rounds) of the education

sector

Rapid literature synthesis

Several broader implementation factors (‘pre-conditions’) at the school level were initially identified, through stakeholder workshops and later verified through a rapid literature synthesis, as being necessary to support effective evaluation systems. The literature reviewed and findings from stakeholder workshops demonstrate a large degree of overlap with the preconditions and the factors stated in the ATPDF. The literature also indicated that effective evaluation systems, those that promote the professional growth of teachers, require four groups of conditions being present: shared clarity of purpose and explicit criteria, relational trust, supportive school leadership and climate, and an evaluative mindset across the whole school to engage in evaluation practices.

The notion of preconditions also reflects a consistent message that is evident in the ATPDF, which is that there are certain factors that must be present at the school level to support teacher learning to flourish. The process of ensuring that the preconditions for evaluation systems are in place also has the potential to change the climate in schools by increasing the level of relational trust within the schools.

(2) principles for selecting evaluation resources

Principles for selecting evaluation resources were identified throughout the study. These principles included the selection of tools and conducting evaluations in general. These principles are intended to guide the practices that should occur within a system containing the necessary infrastructures to support teachers’ professional growth. Across the reviewed literature sources, there was a high degree of convergence in the principles identified. The literature synthesis generated an initial list of principles, which was later tested in subsequent data collection activities. The final 12 principles are presented below in the order of relative importance:

Principle Description

Understandable Assessment, evaluation or feedback must be understandable to the teacher being evaluated.

Professional growthAssessment, evaluation or feedback must be conducted with an intention to support professional growth and must generate information that can support professional growth.

Evidence-based (Assessor expertise)

Assessment, evaluation or feedback must be carried out by professionals with the relevant expertise in the skills being assessed with a high level of knowledge of the expected standards. Further judgements made in the assessment, evaluation or feedback must be justified based on relevant evidence.

ComprehensiveAssessment, evaluation or feedback should involve multiple samples, or multiple assessors and multiple methods. Further evidence can be direct, indirect or supplementary (from a third party).

Authentic

Assessment, evaluation or feedback should occur within the context or setting of the practice of the person being evaluated to capture accurate evidence of skills, attitudes, and knowledge. Evidence gathered needs to reflect real-life practice and needs to be collected in a manner that ensures honesty and validity.

RelevantAssessment, evaluation or feedback should be based on a diverse set of profession-specific standards, competencies or performance metrics which should be identified in relation to the context of practice.

Context specificAll assessment, evaluation or feedback (regardless of tool or resource) should be applicable and where appropriate, tailored to practice settings, individual teacher’s priorities, and school priorities, and contextual characteristics.

Collaborative

Assessment, evaluation or feedback should involve high levels of communication, reflection, and reinterpretation of performance. May include collaboration between the assessor or school leader and the practitioner to enable validation of judgements made or may involve collaboration between peers.

Student-based Assessment, evaluation or feedback should be designed to gauge and measure student growth, engagement, interaction, enjoyment or understanding. Approaches to capturing this might involve students in the feedback/evaluation process, and/or measurement of student growth or entail

Teaching Practice Evaluation Framework | Centre for Program Evaluation x

observation of classroom interactions.

RegularAssessment, evaluation or feedback should be regular, which may involve accumulating data or evidence about performance over a period of time. Regular assessments may provide ongoing progress monitoring and yield feedback on a variety of specific practices.

Professional agencyAssessment, evaluation or feedback domains should be self-directed by the teacher based on self-identified strengths, areas of development, and interests, rather than being compliance-based or solely driven by other school improvement agendas.

UtilityAssessment, evaluation or feedback should be tailored to meet identified areas of professional growth and learning, inform growth opportunities for career development, and/or as aligned with school improvement objectives.

*The principles of utility and professional agency emerged after the first Delphi survey, hence could not be included in this analysis.

An Implementation Plan: The Evaluation Process ModelThe results of the triangulated key findings of the (i) preconditions for building a system that promotes teachers’ professional growth, and (ii) principles for selecting evaluation resources were operationalised into an implementation plan, referred to as the ‘Evaluation Process Model’ (see figure overleaf).

The model is intended for a school audience, where teachers and leaders have a role in the evaluation process. The model provides a suggested scaffold to guide thinking for educators to prioritise, plan, evaluate, and synthesise information. There are four stages in this model:

Phase 1: Focussing

In this phase, the school leader initiates or reviews existing processes and works with teachers to define the purpose and processes of conducting an evaluation of teachers’ professional strengths and growth needs, ensuring that the pre-conditions for conducting the evaluation are in place.

Phase 2: Initiating

Teaching Practice Evaluation Framework | Centre for Program Evaluation xi

Non-metric multidimensional scaling analysis was used to identify the relationship between principles, and the degree to which they are distinct from each other.

The findings indicated there are three higher-order dimensions into which the principles can be grouped:

(1) Content: Principles in this grouping (relevant, understandable, authentic, context-specific) describe the characteristics of content that should be captured in an evaluation, assessment or feedback resource that could be used effectively in a system that supports professional growth.

(2) Purpose: Principles (collaborative, evidence-based, professional growth) that describe the intentional characteristics of evaluation, assessment or feedback that would support teacher professional growth.

(3) Process: Principles (regular, student-based, comprehensive) that describe the characteristics of conducting evaluation, assessment or feedback in a way that would support teacher professional growth.

Altogether, the combination of these dimensions is aimed at enhancing teacher practice and student outcomes, both of which can be considered the product of the evaluation, assessment or feedback event.

Grouping the principles by these dimensions enables a focus on key priority areas for school leaders and teachers when selecting and planning for evaluations to promote teacher professional growth.

*utility and professional agency were

In this phase, the school leader utilises or modifies existing processes within the system to establish the infrastructure and resources in order to create a plan for implementing the process.

Phase 3: Evaluating

In this phase, the school leader and teachers conduct the evaluation: they gather, collate and analyse information collaboratively by following the agreed processes. Consequently, a judgement is made about the subject of the evaluation. This could include clarifying teachers’ existing strengths and identifying teachers’ professional growth needs, progress, new goals, and next actions to achieve those targets.

Phase 4: Supporting learning

In the final phase, teachers and school leaders critically reflect upon not only the processes that were followed, but also on the decision/s made. This feedback is considered and should inform any modifications to processes, infrastructure, and resourcing at the school and system level. To ensure that this process supports system-level learning, strategies are put in place to share the lessons learned.

Teaching Practice Evaluation Framework | Centre for Program Evaluation xii

Figure 1. The Evaluation Process Model

Importantly, the model is built on the foundation of educational program evaluation standards of feasibility (all evaluations should be realistic, efficient, diplomatic and prudent), utility (all evaluations should meet and address the information needs of the end-user audiences), propriety (all evaluations should be conducted in accordance with relevant legal, ethical standards and codes of practice, with the utmost regard for the welfare for all involved in the evaluation), and accuracy (all evaluations should gather and utilise technically adequate information about the components that provide evidence of practice) (Yarbrough, Shulha, Hopson & Caruthers, 2010). Furthermore, the cycle should facilitate bidirectional information flow, feedback, and continuous refinement to enable teacher, school, and system-level professional growth.

Teaching Practice Evaluation Framework | Centre for Program Evaluation xiii

The FrameworkThis document, referred to as ‘The Framework’, was developed based on the findings of the Teaching Practice Evaluation Framework study. It presents the outputs of the study:

1. Evaluation process model

The Framework outlines a process model of evaluation, detailing four phases and underlying processes within each phase. This is intended as a resource to assist educators, including all system stakeholders, in prioritising, planning, evaluating, and synthesising information about supporting teachers’ professional growth. Links to relevant and useful resources have been provided, and a vignette has been developed to apply each of the steps outlined above to contextualise how this process of evaluation could occur in schools.

2. Rubric for identifying and selecting resources

A rubric has been developed to assist educators in choosing appropriate and effective evaluation tools and resources. This was created by utilising the list of principles generated throughout the study. These formed the development of 12 rubric descriptors, which are rated on a four-point scale. The rubric is developmental in nature, allowing users to assess how appropriate an evaluation tool is, considering the professional development needs of the relevant educators.

The rubric is situated within this scaffolding model, where it serves as a practical guide for identifying and selecting resources, alongside other considerations for an evaluative system such as infrastructure, available resources, and plans for implementation.

It is recommended that discussions around tool and resource selection, and use of the rubric, should occur only after the first phase of the evaluation process has taken place. The selection of an appropriate tool (and therefore use of a practical rubric to inform selection), should occur during the second phase in which the infrastructure, available resources, and plans for implementation are considered.

The goal of the rubric is not to simply provide a summative ‘rating’ of a tool; rather, the intention is to assist teachers and school leaders to generate information about the capacity of the tool or resource to be used in the most meaningful way by the relevant parties. Accompanying the rubric is a practical resource providing prompts for different user groups (teachers, school leaders, and the system) to consider alongside principles for selecting tools and resources throughout each phase of the evaluation.

The suggested Evaluation Process Model and rubric will require further development and testing to ensure it is appropriate and useful for schools.

Closing Comments This summary has provided an overview of the study methodology, data collection activities, and findings of the Teaching Practice Evaluation Framework study. In addition, insights into the development of a model have been provided (the Evaluation Process Model). This four-phase model is designed to support schools to operationalise the ATPDF and support the implementation of teacher evaluation in a way that can support the development of teaching practice.

Further testing and refinement through applications of the Framework are necessary to develop bespoke resources and case examples to support widespread uptake and refinement of the Evaluation Process Model and the rubric for appraising evaluation tools and resources. It is our view that the most effective way to do this will be to partner with those educators working in schools, and, importantly, state and territory education departments, regulatory authorities, education unions, and professional associations. The widespread participation of stakeholders from different parts of the system demonstrated the appetite for effective teacher evaluation in Australia, and further, ensured the Framework is based upon a variety of

Teaching Practice Evaluation Framework | Centre for Program Evaluation xiv

views about teaching and the provision of quality education. Continuing to engage in wide consultation in the testing of the Framework would only serve to further maximise its applicability across the diversity of educational contexts in Australia.

To ensure that implementation of teacher evaluation to promote professional growth can occur, it is suggested that further work, support and promotion is needed across several areas to guarantee that the success criteria detailed in the model overleaf can be achieved.

Figure 2. Success criteria for the implementation of the Evaluation Process Model

Teaching Practice Evaluation Framework | Centre for Program Evaluation xv

SYSTEMSupport flexibility for context (e.g. rural and remote)Support adaptable & high quality trainingConsistent communication & use of shared languageSupport and use dataGenerate cost-effective resourcesSupport provision of evaluationMindset shiftSupport implementation of relevant policy

LEADERSHIP & SCHOOLCapacity buildingBuy-in from leadershipInfrastructure to support implementationCost-effective resources given priorityConsistency in deliveryData useContinuity of deliveryAppropriate conditions in placeMindset shift

TEACHERSEngagement in the processBuild capacity to think differentlyUnderstand capable learnersMindset shiftBuy-in from teachersGather data & communicateCost-effective resources given priority

Teaching Practice Evaluation Framework | Centre for Program Evaluation xvi

The recommendations for implementing an evaluative system are:

1. Adopt an evaluative thinking mindset

A mindset shift is needed towards prioritising evaluative thinking and making it commonplace, which would empower all educators within the system. Buckley, Archibald, Hargraves and Trochim (2013) proposed the following succinct definition: “Evaluative thinking is critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective-taking, and informing decisions in preparation for action.”

2. Teacher assessment and evaluation for professional growth needs to generate actionable feedback

The model and resources for teacher evaluation need to clearly specify and guide teacher assessment and evaluation that generates actionable, relevant, and useful feedback for professional growth. The information generated through assessment and evaluation needs to enable and inform evidence-based practice change in alignment with an individual’s professional growth goals.

3. Implementation needs to be considered as an area of expertise and valued

Stakeholders in this study raised several implementation factors that are important to consider when planning for teacher evaluation. School leaders need to take responsibility for ensuring the preconditions are in place to enable teacher assessments and evaluations to occur and teachers need to increasingly ensure they see themselves as active agents in the process. Teachers and educators also need to have the time to reflect on their goals and trial new or different evaluation practices to find what works for them, in their context and circumstances.

4. Evaluation capacity strengthening, resourcing and promotion

Any reform and its ensuing implementation needs resourcing. Support at all levels of the education system is pivotal to the development of an evaluative model. This study demonstrated that a mindset shift is necessary along with the development an effective and efficient infrastructure to support implementation of an evaluative model or system. This support needs to take the form of building evaluative capacity of both teachers and leaders, promotion of the value of evaluative thinking as well as promoting the merit and worth of the model or system and finally, strengthening the availability of quality evaluation resources.

5. Practices should be teacher-driven and supportive of professional autonomy

It is necessary for teacher assessment and evaluation practices to be teacher-driven, rather than solely focused on school improvement. This goes beyond the common goal of supporting ownership and speaks more to what it means to be a teacher as a professional. Teachers need to know that they are able to take responsibility for their assessment and evaluation, and they need to be the stewards of the evidence and of monitoring their professional growth, in shared partnership with school leadership and the system. Teacher assessment and evaluation for professional growth must be supportive of professional autonomy.

6. Information flow should facilitate data sharing at every level of the system

The Evaluation Process Model should support the development of policy and implementation models in various contexts. This would enable rich data and provide a basis for information sharing across jurisdictions, sectors, and school contexts, and further support the goal of educational reform so that it is authentic and evidence-informed.

Information flow should always facilitate a feedback cycle at every level of the system, such that an evaluation activity with a single teacher becomes a part of informing the national educational reform agenda through its collective impact about how to support teacher professional growth in Australia, and in so doing, improve the quality of education for all.

7. Consider Collective impact

There is a necessity for the shared roles and responsibilities at all levels in this evaluation system. Clarity of these shared responsibility is essential (e.g. system to support notion of integrating use of evaluation, creating usable resources for the profession). Considering the idea of Collective Impact may be valuable (Kania & Kramer, 2011).

Part A: The Evidence for Promoting Teachers’ Professional Growth1.IntroductionIn November 2018, the Australian Government Department of Education and Training (DET) commissioned the Centre for Program Evaluation (CPE) to develop a framework to inform evaluation to support the professional growth of teachers.

This study builds on the findings of two previous studies commissioned by the DET and conducted by CPE: Teacher Effectiveness Systems, Frameworks and Measures: A Review (Clinton et al., 2016), an extensive international review of teacher evaluation systems, frameworks and measures to identify practices that promote teacher practice improvement through evaluation; and Investigating the Key Characteristics of Effective Teachers: A Systematic Review (Clinton, Aston, & Koelle, 2018), a review of secondary research on the individual characteristics of effective teachers that directly or indirectly influence student, school and teacher outcomes. These studies in combination are designed to support evaluative thinking across schools and systems, while providing access for teachers and schools leaders to a coherent, evidence-informed process of how to enrich an educator’s professional journey.

The previous work has clarified what effective systems and processes for teacher evaluation look like, along with defining the assessable factors that comprise the nexus between teachers, effective teaching practices, and student learning outcomes. With the Teaching Practice Evaluation Framework, as the third piece of this tripartite body of work, the aim is to take a consultative approach to develop a useful, practical, and implementable Framework. Ideally, the Framework will be used to guide educators to implement practices and select robust resources that are known to facilitate teacher effectiveness and professional growth and development, ultimately improving student learning outcomes.

This report proceeds in two sections. The first section (Part A) is a foundational document that details the evidence and conclusions in relation to building such a system along with recommendations for implementation. The second section (Part B) presents a practical guide to promote the use of the Framework.

1.1. Background The quality of teaching has been shown to produce the greatest impact of all in-school variables on student performance (Hattie, 2008). In fact, teaching quality is more predictive than students’ socioeconomic backgrounds (Darling-Hammond, 2000). Given this, the effect of poor quality teaching can be cumulative and debilitating on student outcomes (Darling-Hammond, 2000). Slater, Davies, and Burgess (2012) noted that students displayed almost a 12-month difference in academic progression when taught by a low-performing teacher compared with a high-performing teacher.

Effective evaluation of teachers can enable continuous and targeted professional development while allowing educators to reflect on quality evidence and develop or adapt their practice. There is growing evidence that evaluative practices by educators are essential for school improvement (Earl & Timperley, 2015; Hattie & Timperley, 2007). Further, highly visible teacher effectiveness measurement and evaluation have also provided opportunities to incentivise, recognise, and reward teaching abilities and high performance. This, in turn, can help support the status of teachers as professionals in Australia, which can have a positive influence on teacher quality by attracting highly capable pre-service teachers, who become professionals that are more likely to engage in appropriately-targeted ongoing professional learning and professional growth activities.

At a system level, both nationally and internationally, governments and policy makers have recognised that teacher and teaching quality are the keys to improving educational outcomes for students. Generally,

Teaching Practice Evaluation Framework | Centre for Program Evaluation 1

teacher effectiveness can be conceptualised as a spectrum covering the extent to which a teacher is able to progress student outcomes. This is often measured using student achievement results and other summative tools. Teacher quality, however, as demonstrated by previous reviews, also includes teacher attributes, such as capabilities, training, knowledge, and beliefs (Clinton et al., 2016; Clinton et al., 2018). Both teacher and teaching quality have become critical components of school improvement and there has been considerable debate regarding their definition and appropriate use.

In Australia, in the last decade, supporting teacher effectiveness has become a cornerstone of several large-scale education reforms, including the Australian Professional Standards for Teachers (APST), together with processes outlined in the Australian Teacher Performance and Development Framework (ATPDF) (Australian Institute for Teaching and School Leadership, [AITSL], 2013). The ATPDF defines the evaluation of teacher performance as an ongoing cycle and provides a structure for appraising, developing, and improving teaching practice. The framework also articulates that it is essential for teachers to receive meaningful feedback and support to bring about long-term career progression and enhanced levels of student achievement (Black & Wiliam, 2009; Clinton et al., 2018; Hattie & Timperley, 2007). However, the literature has highlighted the gap between policy and practice in teacher evaluation for the purposes of ongoing professional development, as there is limited evidence on quality teacher evaluation practices (Clinton & Dawson, 2018; Freeman, O’Malley, & Eveleigh, 2014). This is compounded by a general lack of agreement on how teachers’ professional learning needs should be evaluated, along with a lack of clarity around the selection of teacher evaluation tools to ensure fit-for-purpose assessment.

The Australian Teacher Performance and Development FrameworkHistorically, Australian states and territories have often adopted disparate approaches to teaching and learning (Freeman et al., 2014). Given that responsibility for teacher evaluation predominately lies with the states and territories, there is little consistency or presence of a national approach to teacher evaluation (Clinton et al., 2016; Clinton et al., 2018). The introduction of the APST and the ATPDF progressed Australia further towards a nationally consistent approach to teachers’ career progression and professional learning. However, the detailed nature and degree of implementation of the ATPDF in each state and territory is not known. Given that states and territories along with different school sectors often detail their own frameworks for performance and development reviews, variance in the adoption of the ATPDF is likely. Furthermore, while it is considered good practice for review and evaluations to be conducted on an annual basis, not all jurisdictions have mechanisms in place to ensure that these occur (Clinton et al., 2016). The table below highlights where there are known gaps in knowledge about teacher evaluation in Australia. It should be noted that the information in the table draws from the two previous reviews conducted (Clinton et al., 2016; Clinton et al., 2018) and the stocktake of teacher evaluation tools and practices conducted by AITSL (2018b). The key findings of which are discussed further in the following section.

Table 1. Implementation of the Australian Teacher Performance and Development Framework Across the System

Component of implementing ATPDF National State/Territory Regional Local school

Policy and strategy ✔ ✔ Unknown Unknown

Mindset ✔ Unknown Unknown Unknown

Context & culture ✔ Unknown Unknown Unknown

Standards ✔ ✔ ✔ ✔

Implementation plan Unknown Unknown Unknown Unknown


Conditions for implementation ✔ Unknown Unknown Unknown

Resources to support implementation ✔ ✔ Unknown Unknown

Analysis of information Unknown Unknown Unknown Unknown

Use of information Unknown Unknown Unknown Unknown

As noted in the table, there are considerable gaps in information relevant to the implementation of the ATPDF (and teacher evaluation more broadly) in many areas, particularly at the regional and local school levels. The literature and AITSL’s stocktake of teacher evaluation tools highlighted the variance of evaluation practices at the school level. While this means that approaches to evaluation are adaptable to the contexts of different schools and their improvement agendas, it also perpetuates inconsistency in the implementation of evaluation and sharing of evaluative data. A lack of consistency in the implementation of evaluation systems and practices across different contexts can reduce the potential impact of evaluation on teaching and learning (Stronge, 2005). To contribute to school improvement at all levels of the system, a consistent approach to the development of a culture of evaluation, and, critically, the use of evaluative information across the Australian education system would be advantageous.

The Status of Teacher Evaluation in AustraliaThe two previous reviews commissioned by the Australian Government Department of Education and Training, conducted by the Centre for Program Evaluation (Clinton et al., 2016; Clinton et al., 2018), synthesised evidence to develop a knowledge base of what an effective teacher evaluation system should entail, including the use of tools and measures to support teachers in continuously improving the characteristics and behaviours known to have a positive impact on students. Key findings for each are provided below.

Teacher Effectiveness Systems, Frameworks and Measures: A Review The first review, Teacher Effectiveness Systems, Frameworks and Measures: A Review (Clinton et al., 2016), examined teacher evaluation systems, frameworks, and measures within Australia, New Zealand, England, Scotland, Germany, Austria, Singapore, Hong Kong, South Korea, USA and Canada for their transferability to the Australian education context. The specific aim was to review and synthesise current international policies and practices regarding teacher evaluation and evaluate these systems to subsequently inform the development of a teacher effectiveness measure for Australia.

This review involved a rapid systematic synthesis of peer-reviewed research, grey literature, and policy documents relating to teacher effectiveness and evaluation procedures. The review demonstrated that teacher evaluation processes and measures of effectiveness range from sophisticated national structures to informal approaches. This descriptive exploration into systemic variables found that effective systems of assessment are differentiated, with multiple modes informing teacher practices and evaluators who are well trained. In addition, these approaches to evaluation are framed by systemic and structural standards, including professional standards, which govern teacher practices. The evaluation of teacher practices is also supported by a wide range of tools and methods, such as classroom observation, self-appraisal, teaching portfolios, and performance interviews (OECD, 2013).

While the report reaffirmed that quality teaching practices within the dimensions of pedagogical content knowledge and reflective practice are related to student achievement, it also highlighted that there is an important distinction between teaching quality (practice and instruction) and teacher quality (personal attributes, competency, skill and dispositions; Darling-Hammond, 2012). Further, the findings of the synthesis suggested that the dimensions of effective teaching can be organised into four categories: a) teaching; b) the teacher as a person; c) behaviour as a teacher; and d) continuous learning. The variables within each category are detailed in Figure 3.


Figure 3. Teacher and teaching factors that relate to educational impact.

To assess and evaluate teaching effectiveness within and across these four categories of variables, the review recommended that the assessment and evaluation of teacher effectiveness be considered as a multi-dimensional construct with a range of tools and measures to ensure reliability and validity within regulatory standards and should be fit-for-purpose.

To implement and embed such evaluation practices, the review highlighted the need for teacher evaluation tools to be grounded within evaluation systems and accountability structures. The latter is important for ensuring that evaluation processes can be systematically incorporated into systems for registration, professional development, and, where necessary, performance development. Some of the major features of effective teacher evaluation systems include:

A coherent evaluation framework Multiple evidence-based measurement dimensions Multiple evidence collection methods and tools, and A well-trained evaluator workforce.

The next section expands on the second review, Investigating the Key Characteristics of Effective Teachers.

Investigating the Key Characteristics of Effective TeachersWhereas the first review, Teacher Effectiveness Systems, Frameworks and Measures: A Review (Clinton et al., 2016), addressed how teacher practice is evaluated internationally and what features of teacher practice relate to student outcomes, the second review, Investigating the Key Characteristics of Effective Teachers, examined the features of teachers as individuals that have been empirically shown to influence student and school outcomes.


In relation to Figure 3, the domain of teacher variables within the intrapersonal realm has received considerably less attention in the literature compared with other domains despite their potential to influence teachers’ behaviours and effectiveness in the classroom. Intrapersonal teacher characteristics can act as mediators and therefore can be thought of as proxy indicators for other variables associated with student growth (Muñoz & Chang, 2007). The second review entailed a systematic review aimed at identifying the key characteristics of effective teachers and examining the magnitude of their contribution to school, student, community, and teacher outcomes, and providing further depth to teacher evaluation in terms of domains and scope of assessment for the teacher as an individual.

Of the 11 173 articles that were screened, 52 secondary studies that met the inclusion criteria were reviewed (a total of 1 427 primary studies were represented in the secondary research studies).

Ten characteristics of effective teachers directly or indirectly related to student, teacher, and school outcomes were extracted and defined from the 52 studies: cognitive capability; social and emotional learning/competence; self-efficacy; communication; motivation; cultural competence; attitudes, beliefs and expectations; personality type and traits; self-reflection; and collegiality. Further, ‘values, morals, and ethics’ were included as the eleventh characteristic from a consideration of a sub-sample of sources.

These 11 characteristics were then collated and related to one another in an explanatory model, which was workshopped with DET and AITSL. This is reflected in Figure 2.

Figure 4. Explanatory model of effective teacher characteristics (Clinton, et al., 2018).

The explanatory model above elucidates the conceptual relationships between the teacher characteristics and behaviours that were empirically shown to influence student, school, and teacher outcomes.

As illustrated in Figure 4, the 11 characteristics were conceptually delineated into trait, state, capability, and behavioural dimensions. This delineation was based on the following definitions.

Table 2. Dimension Definitions in Explanatory Model of Teacher Characteristics

Dimension Definition/constitutive elements

Trait Trait characteristics are those personality variables that relate to extraversion, openness,


Dimension Definition/constitutive elements

conscientiousness, neuroticism, and agreeableness.

State State characteristics are temporary behaviours and/or feelings that depend on an individual’s situation and motivations at a particular time.

Capability Capability characteristics refer to the individual’s capacity to engage in evaluative thinking, cultural competency, and the commitment to and motivation for (teaching) practice.

Behavioural Behaviours refer to the ways in which an individual (teacher) communicates, their level of collegiality, e.g. their collective efficacy working within teams and organisational (school) contexts; and finally, the professional image and competence they project.

The model suggests that the dimensions of trait, state, and capability relate to teaching, and consequently shape a range of individual teacher attitudes, beliefs, and expectations. Further, the explanatory model suggests that the aforementioned factors also mediate the practices of teachers.

As part of the Investigating the Key Characteristics of Effective Teachers review and in collaboration with AITSL, the 11 characteristics were mapped against the APST (AITSL, 2017). The characteristics identified were found to relate closely to the Standards. Specifically, the characteristics reflect the behaviours teachers would need to exhibit when practising in a way that aligns with the Standards.

Evaluation Tools Used in Australian Schools At the time of conducting the Investigating the Key Characteristics of Effective Teachers review, AITSL was engaged in undertaking a stocktake of the use of teacher evaluation tools in Australian schools and classrooms. Among the participating AITSL educators, reported teacher evaluation tools were grouped into four categories: (1) AITSL/National teacher certification resources; (2) Professional Growth Network resources; (3) tools outlined in Teacher Effectiveness Systems, Frameworks and Measures: A Review (Clinton et al., 2016); and (4) selected teacher effectiveness evaluation tools/strategies.

The tools (see Appendix D) were reported, at the time of the review, as being used in a sample of Australian schools and classrooms. Sources of teacher practice data for each tool (e.g. from teachers, peers, leaders, students, and artefacts) varied depending on the nature of the tool itself and prescription of self, peer, and external reporting mechanisms. The tools listed in Appendix D were used for both summative and formative purposes, with varying psychometric properties in relation to their overall validity and reliability. Further, at the time of the review, it was not known how the data gathered from these tools were used by stakeholders. Thus, the review concluded that these tools and the evidence gathered from their implementation varied across jurisdictions and school sectors.

The targeted review of teacher evaluation tools illustrated two key conclusions of relevance for this study:

(i) The evaluation of teachers in Australia is not clear or uniform; it is not known how often teacher evaluation tools are implemented and how the data gathered from these tools are used within different jurisdictions and school sectors; and

(ii) the current state of teacher evaluation in Australia lacks a systematic and implementable framework to guide the selection of teacher evaluation tools.

Thus, there is a clear policy implementation gap demonstrated by AITSL’s stocktake and the findings of the two prior syntheses. There also appears a gap in the development or presence of system structures to support effective and consistent teacher evaluation in Australia. Accordingly, the Australian Government Department of Education and Training commissioned the study to undertake wider stakeholder


consultation and develop a ‘Framework’ to help address the policy implementation gap and inform the development of supportive system structures for teacher evaluation.

Study AimsThe study aimed to incorporate the findings from previous reviews (Teacher Effectiveness Systems, Frameworks and Measures [Clinton et al. 2016], and Investigating the Key Characteristics of Effective Teachers [Clinton et al, 2018]) into an implementable framework for conducting teacher evaluation in Australia. A key objective of the study was to ensure that it incorporated the voice of the education profession; that is, both end-users and practitioners—with a special emphasis on school-level teachers and leaders. The overarching aim was to develop a culture of evaluative thinking with teachers, leaders, schools, and systems. The intended outputs (with practical resources) of the study include:

To develop a process for implementation of an evaluative system, and To understand the conditions necessary to implement the system and understand appropriate use of

tools.

Figure 3 depicts how the study was designed to build upon the findings of the previous studies.

Figure 5. Design of the study.

2.MethodologyThe overarching design of the study and the key activities completed between February–September 2019 are discussed in this section. The study was underpinned by a mixed methods research design, where both qualitative and quantitative data were collected through several stakeholder consultation methods, along with rapid synthesis of relevant literature. Creswell and Poth (2017) suggested that this mixed method approach involves collecting “either simultaneously or sequentially both numeric information (eg instruments) as well as text information (eg interviews) so that the final database represents both quantitative and qualitative information” (p. 18). The key activities and their relationship with one another are articulated in the figure below.


Figure 6. Phases of the project.

A detailed description of the technical procedures (data collection and analysis) of the literature scans, development and administration of the modified Delphi survey, and stakeholder consultations (interviews and workshops) is provided in the following sections.

2.1. Literature ScansThe literature scans were designed to provide an overview of the contemporary evidence in relation to several areas: a) implementation of evaluative systems in schools; b) understanding the preconditions and necessary infrastructures for effective evaluative systems; and c) selecting and using available resources. The scans were not designed to be exhaustive; however, they did provide evidence through secondary sources of the current thinking and foundational view of the evidence. The first stage of the methodology involved a comprehensive literature scan. The table below shows the criteria and framing for literature that was included in the scan. The purpose of examining sectors outside education was to gain a more comprehensive understanding of professional development frameworks that focus on supporting teachers’ professional growth.

Table 3. Literature Scan Search Criteria

Search Criteria Description

Publication date Limit to published literature within the last 5 years

Sectors Human service professions, including teachers, social work and justice, allied health, clinical health, defence and army, emergency service personnel

Eligible studies Any of the following that include guidelines, criteria, frameworks or tools for professional practice evaluation:



Policy documents on existing evaluation tools

Theoretical and/or commentary articles on evaluating/assessing personnel evaluation tools

Existing reviews of personnel evaluation (narrative syntheses, systematic reviews and meta-analyses)

Psychometric studies on the quality and validity of personnel evaluation tools

Qualitative studies on the quality and validity of personnel evaluation tools

Countries Studies conducted in OECD countries

Language Studies in English language

The search strategy to source literature meeting these inclusion criteria is provided in Appendix A. The literature scan process encompassed five stages:

1. Searching electronic databases, grey literature, and additional hand-searching in specific journals or websites

2. Snowball searches of articles3. Source screening by title and abstract. A team of reviewers conducted title and abstract screening of

sources using inclusion and exclusion criteria. Reviewers marked sources as ‘Relevant – proceed to full-text screening’, ‘Non-relevant’, or ‘Unsure’. A second reviewer resolved sources marked as ‘Unsure’. Reviewers noted reasons for inclusion or exclusion in full-text review stage.

4. Full-text screening. Reviewers conducted a full-text screening of sources to assess source relevance and quality. Sources deemed eligible progressed to the data extraction stage.

5. Data extraction. Relevant guidelines or criteria with applicability to the service profession or the broader education sector were extracted from the sources.

The sources that were reviewed in the literature scan included published journal articles, organisational and policy documents, and grey literature describing evaluation frameworks in education as well as in other sectors of interest. Sources included factsheets, rationale for assessments, and administration of self-assessments based on identified competencies. Employing the search procedures described above and using the search strategy provided in Appendix A, a total of 133 sources were selected as relevant for full-text review.

Reviewed documents identified from multiple sectors included guides for the evaluation of competency, capability, or readiness for registration, procedures for conducting professional development assessment, tools (both formative and summative) for assessing competency, fact sheets for performance assessment and delivery, administration of such tools, and documents that support personnel to self-evaluate based on existing competency frameworks and standards.

Overall, the sources reviewed from these scans were primarily personnel evaluation and professional development frameworks in the education sector, along with other human service sectors, including nursing, social work, allied health, police and medical education professionals. Where available, any guidelines or criteria that underpin such frameworks were also reviewed, eg criteria used to grant certification/accreditation or promotion.

2.1.1. Analysis ProceduresAn inductive content analysis approach, as described by Schreier (2012), was undertaken on data extracted from reviewed sources, where common themes were generated directly as a result of thematic analysis.


Several personnel evaluation and practice frameworks from other sectors, ie allied and clinical health, helped to inform this process. The figure overleaf shows the steps undertaken in order to derive the initial list of proposed principles and descriptors for this study.

Figure 7. Literature scan data extraction and analysis process.

The output of the literature scan was a list of principles of practices and resources that promote professional growth and the preconditions required for such principles to be enacted. The generation of these principles was informed by extracting information from reviewed literature that related to:

How to assess, review and/or conduct personnel evaluation When to use the evaluation tool and where How to identify fitness-of-purpose of evaluation tools Target audience (type of practitioner) Use of information gathered through the evaluation.

A summary of findings from the literature scan, including the principles, is provided in the section 3.0.

2.2. Stakeholder Consultations In conjunction with the literature scan, consultations with stakeholders also took place in collaboration with AITSL. As described below, while the consultations (separate from workshops described later) were opportunistic, they were all conducted with educators who represent the end-user audience(s) for the Framework. Several group consultations with professional bodies and experts were also conducted. The output of these activities was a set of 11 principles1, which formed the basis for the design of an online modified Delphi survey.

2.2.1. Group ConsultationsThe project team conducted two group consultations with stakeholders from various jurisdictions, professional practice and leadership expertise specialisations, and educational organisations. These consultations drew upon the expertise of practising educators, education leaders, and policy makers to inform understandings of the principles developed through the literature scan.

1 These later became 12 principles based on the findings of subsequent consultations and the modified Delphi survey.


The first group consultation was with an existing cross-sectoral national consultative group which consisted of expert educators2 (n=16), who were either Highly Accomplished and Lead Teacher (HALT) classroom teachers, or pedagogical leaders that had been or were presently involved in the development of assessment, best practice understandings, and policy guidelines for education in Australia.

The second group consultation was with a peak body whose members are national professional teacher associations of different school types, subject disciplines and localities (regional and remote). Attendees (n=17) represented a diversity of curricula and practice areas through their associational affiliation. Curricula subject areas included environmental studies, computer science, history, social and citizenship, business and literature studies. Other participants represented associations with interests in the practice surrounding the education of gifted and talented students and the provision of education in regional, rural, and remote communities.

During the consultations, the project team posed structured questions to elicit viewpoints on current teacher evaluation practices and methods, including the following:

How do you know what your teachers’ professional learning needs are? What do you do to support teachers’ professional learning? How would you know that what you do is supporting your teachers’ professional learning needs?

In addition to these questions, the project team also asked participants to complete a short ranked-item questionnaire via an online polling method (pollev.com). Specifically, each participant was individually asked to rank, in the order of importance, the principles identified from the literature scan. The objective of this exercise was to prompt participants to carefully review the principles and identify missing elements that they deemed necessary or important for evaluating educators to support professional growth.

The findings from the group consultations informed refinement of the principles prior to the modified Delphi survey.

2.2.2. Stakeholder InterviewsSimultaneous to conducting the group consultations, a small number of semi-structured consultation interviews were conducted with international experts in teacher evaluation, teacher assessment, and performance and development assessment in other human service sectors. In total, one focus group session and several individual interviews were conducted.

The purpose of these semi-structured interviews was to:

Understand the culture of evaluation of the professions Understand the standards for evaluation across professions Gather perspectives on the list of principles for utilising resources Gather perspectives on evaluation from other human service sectors Identify necessary preconditions for conducting personnel evaluation for the purposes of ensuring

professional growth Gather feedback that reflects international developments in teacher evaluation.

Interviewees were identified from contexts explored within the scope of the Teacher Effectiveness Systems, Frameworks and Measures review (Clinton et al., 2016). This included England, Scotland, and the USA, based on similarities to the Australian context and the comparability of education systems (Clinton et al., 2016). Contacts for stakeholders from professional bodies and sectors, such as police, nursing, and allied health and medical education professions, were also invited to participate in the interviews.

2 Organisational affiliations represented in the composition of this group include the Australian Secondary Principals Association (ASPA Ltd.), Australian Institute for Teaching and School Leadership (AITSL), Grattan Institute, National Advisory Council of Advanced Practice Provider (APP), and the Australian Children’s Education & Care Quality Authority (ACECQA).


The information gathered in the stakeholder interviews was used to develop a refined list of pre-conditions and principles, with associated definitions for appraising teacher evaluation tools, ready for testing through stakeholder workshops and the modified Delphi survey.

2.2.3. Stakeholder WorkshopsFrom June–July 2019, four structured stakeholder workshops were conducted to gather the perspectives of key stakeholders representing the education profession. The primary objectives of the workshops encompassed the following:

Gather input from the expertise of the participants in workshops into what teacher evaluation could be used for

Identify the potential uses and purposes for teacher evaluation in different contexts (eg across sectors or states and territories)

Identify considerations for how the educators currently select and implement assessments for teachers’ professional growth

Identify the needs of stakeholders when undertaking teacher evaluation practices Draw upon expertise and insights about teacher evaluation in Australia.

(1)PARTICIPANTSTo identify educational organisations that represented the end-user audience for this study, we gained input on potential workshop participants from both the Australian Government Department of Education and Training and AITSL. The stakeholder sampling process also included a snowballing approach to identify additional groups.

DET and AITSL were provided with an initial list of organisations (n=110) who were viewed as either the primary audience of this Framework as the intended end-users or as practising educators and experts with interest in conducting teacher professional growth and education evaluation in Australia. Further, this list also included an explicit focus on regional, rural, and remote stakeholders.

Based on feedback from DET and AITSL, an additional 21 relevant stakeholder organisations or branches were added. Table 4 lists stakeholder from the organisations, which included teachers and leaders, representatives from teacher unions, teacher employers, policy makers, government departments, teacher professional associations, principal professional associations, researchers and thought leaders, regulatory authorities, and assessment developers.

Table 4. Relevant Stakeholder Groups and Perspectives

Group Perspective

Teachers Union Practitioners & commentators

Teacher employers / policy makers Policy makers, education sectors

Teacher professional associations Practitioners

Principal professional associations Practitioners

Researchers / thought leaders Researchers

Regulatory authorities Policy makers

Assessment developers Researchers / policy makers

Parents associations, special education needs support Inclusive education


An invitation to participate in a workshop was distributed via email to all 131 identified organisations. To more effectively disseminate information and invitations to a greater number of relevant stakeholders, a snowballing approach was also undertaken in which national or peak bodies were contacted to disseminate invitations and relevant materials, and to encourage state and territory counterparts to participate by identifying and communicating with representatives. Each organisation or invitee was asked to distribute the email invitation among their networks or other professional associations.

Ancillary dissemination channels included the newsletters and email databases of the Australian College of Educators and AITSL newsletters to Highly Accomplished and Lead Teachers. Invitations to participate were also promoted via the Melbourne Graduate School of Education social media channels and shared from the University of Melbourne social media accounts.

(2)LOCATIONSWorkshop locations were identified in close consultation with DET. The main goal was to maximise participation from stakeholders across Australia. Initially, three locations were identified: Sydney, Melbourne and Perth. Additional funding was received from DET to conduct a fourth workshop in Darwin to support participations of stakeholders located in more regional, rural, and remote areas.

(3)DESIGNThe structure and activities implemented in the workshops were designed in close consultation with DET and AITSL. A combination of structured small group activities and larger group discussions were utilised to prompt discussions about teacher effectiveness and professional growth, elicit considerations for conducting context-specific professional growth assessments, and define the processes involved in selecting assessment tools that are useful to the formulation of an implementable Framework.

Where applicable, workshops were used as an avenue to explore interim findings from other consultation processes such as the group and expert consultations described in the section above. This was concurrent with the Delphi survey, with the aims of interpreting and substantiating results and identifying where there was convergence or divergence in views. Workshop activities and their associated purpose and aims are described further in Section 3.2.

2.2.4. Modified Delphi SurveyThe Delphi method is a structured and systematic process of gathering data from experts to determine a range of opinions, and to establish consensus in views from stakeholders through an iterative process. While there are a range of methods that can be used to conduct a Delphi study, online surveys are a common approach. This was adopted in the study; specifically, the modified-Delphi survey was designed to obtain views on:

Essential principles and criteria for the selection of tools, Key factors to consider when conducting teacher evaluation, and Key needs (systems, structures) of end-users, such as school-based practitioners to conduct teacher

evaluation.

(1)DESIGNAs per the design of the Delphi method, multiple rounds of the survey were conducted. The design of the survey captured demographic information (eg role, type of organisation, jurisdiction, and sector), in addition to respondents’ views on principles, broader factors, and systems and structures needed to conduct evaluations to support teachers’ professional growth. The design of the first round of the Delphi survey incorporated a list of criteria and definitions developed based on the literature scan (see Section 3.1). Prior to being administered, preview links were shared with DET and AITSL for feedback and review.

A variety of question types, including ranked item exercises, Likert scale responses, and open-ended free-text comments were used in the survey. In the ranked item exercises, participants were asked to rank the


11 principles in relation to other items in order of importance. In the Likert scale section, descriptions of the principles were broken down into individual statements (items), for which users were asked to rate each item on a 10-point scale, where 1 = not at all important, and 10 = extremely important. Open-ended questions were included at the end of the survey, with the aim of gathering rich perspectives on redundant or missing items, participants’ views on teacher evaluation, current practices, and specifically, structures and systems that are necessary to support teachers’ professional growth.

The distribution of items that were ranked on the 10-point scale was assessed after the first round of the survey was administered. The cut-score was determined to be 70 per cent or above of responses rated above 7, with these items meeting this cut-score (or, in other words, where there was a consensus that these items were important) progressing to the second round of the survey. Items which were scored below this threshold after the first round (those that had less than 70 per cent of respondents providing a rating of above 7) were removed for the second round.

Following the analysis of the first-round response, the survey was updated with items below the cut-score removed and new items added. However, participants were asked to ‘endorse’ the removal of items from the Framework, or whether these should be included.

Because the Delphi approach entails iteratively collecting data from a knowledgeable participant pool over multiple rounds of data collection, all participants were invited to participate in the second and final round in July 2019. The survey required participants to use a unique identifier, allowing the project team to identify participants who completed both rounds and to match responses over time.

(2)SAMPLINGKey stakeholders invited to participate in the first round of the modified Delphi survey followed the method utilised to determine appropriate groups to participate in the workshops (see Section 2.2.2). The stakeholder sampling process for the modified Delphi survey also included a snowballing approach, in which invitees were asked to distribute the invitation and links within their networks and organisations, as befitting the intended audience for the survey3. Links to participate were also shared via the Melbourne Graduate School of Education and the University of Melbourne Twitter and Facebook accounts.

The first round of the survey was administered on 16 May 2019 where fortnightly reminders were sent via email to participants to participate or share the links. The survey was closed on 21 June 2019 after a period of five weeks, with analyses of data conducted using responses received during this window.

As per the Delphi method, all stakeholders who were invited to participate in the first round received a follow-up invitation to participate in round two. The survey was also open to stakeholders who had not completed the survey in round one. Recipients were also asked to share the invitation within their networks, with survey links shared via the Melbourne Graduate School of Education and the University of Melbourne Twitter and Facebook accounts and the Victorian Institute of Teaching (VIT) website. The survey was administered on 15 July 2019 and fortnightly reminders were sent via email, or regularly re-posted on social media channels for five weeks until the closing date on 23 August 2019.

(3)ANALYSISThe modified Delphi survey was administered through the online survey manager, Qualtrics. Response monitoring occurred throughout the process to ascertain engagement levels. Data was exported from Qualtrics into Microsoft Excel for cleaning and analysis.

Descriptive statistics were calculated for both survey rounds, where mean scores and standard deviations were calculated for the included survey items at the item and principle level. Following this, interpretive statistics were employed with the explicit purpose of determining the level of agreement among

3 In the invitations to participate in the survey, the specific text stated “school-based teachers and leaders, thought leaders, policy makers, researchers and practitioners”.


respondents. Statistically, these analyses were conducted to determine inter-rater reliability (where each respondent was considered a ‘rater’). Intraclass correlation coefficients (ICC) and Krippendorff’s Alpha Reliability were calculated using the statistical analysis software package R to identify the extent to which there was agreement among respondents about the principles (Gamer, Lemon, Fellows, & Singh, 2013).

Open-ended responses were analysed using an inductive thematic analysis approach (Thomas, 2006). The findings of these analyses are reported in Section 3.3.

2.3. Data Triangulation ProcessFinally, the findings of all data collection activities, literature scan, stakeholder consultations (including workshops), and the modified Delphi survey were triangulated (through matrices) to determine the degree to which the findings were consistent across data sources. Consistency was first examined through the degree to which the findings were similar across sources. Secondly, the triangulation involved assessing the strength of evidence for each key finding across the data sources.

Overall, to develop the Framework, this study involved gathering both quantitative and qualitative data from educators in Australia and internationally, along with conducting additional targeted literature scans. The proceeding section presents and discusses the findings of each of the activities conducted with stakeholders and ends with a section on triangulation findings.


3.Results This section provides an overview of the results and findings of the data gathered. Findings are reported according to the different methods employed in this study: the literature scan, expert interviews, stakeholder workshops, and the two rounds of the Delphi survey. This section ends with triangulation and synthesis of the key findings.

3.1. Targeted Literature ScansAs previously described, a literature scan was conducted for this study. There were three purposes for the scan, namely:

1. Identification of characteristics of performance and development frameworks2. Identification of preconditions for conducting teacher evaluation3. Identification of steps, instructions or guidelines for the selection of teacher evaluation tools and/or

planning of evaluation practices.

An overview of sources reviewed is provided below, followed by a brief discussion of key findings.

3.1.1. Reviewed SourcesEmploying the search procedures described in Section 2.1 and the search strategy provided in Appendix A, a total of 133 sources were selected as relevant for full-text review.

Sources reviewed from multiple sectors included guides for the evaluation of competency, capability, or readiness for registration, procedures for conducting professional development assessment, tools (both formative and summative) for assessing competency, fact sheets for performance assessment and delivery, administration of such tools, and documents supporting personnel to self-evaluate based on existing competency frameworks and standards. Table 5 indicates the spread of sources reviewed by sector.

Table 5. Sources Reviewed by Type and Sector

Sector Sources reviewed (n)

Education 96

Nursing, police, psychology, allied health 25

Multiple 12

Total 133

Relevant data were extracted (based on the three areas of investigation) from reviewed sources. An inductive content analysis approach was undertaken to analyse these data and derive common themes (Schreier, 2012).

3.1.2. Establishing Systems for Teacher Evaluation that Promote Professional Growth

Many countries have teacher evaluation systems to improve teaching practice (eg, New Zealand, USA, England, Scotland). Generally, it appears that there are two main purposes of teacher evaluation: firstly, it is formative and therefore influences professional development; and secondly, it increases accountability for teacher performance (Stronge, 2005), in particular, for impact on student learning. Teacher evaluation systems can play an important role in improving teacher effectiveness (Clinton et al., 2016). By stimulating the professional development of teachers, teacher evaluation systems also have an indirect influence on the results of students (Robinson, Hohepa, & Lloyd, 2009). Professional learning is therefore significantly


related to the worth of an evaluation system for teachers (Stronge & Tucker, 2003). Although teacher evaluation can improve teacher effectiveness and have a positive impact on student outcomes, the design, implementation, and effectiveness of teacher evaluation systems in many educational contexts are lacking (Kyriakides & Demetriou, 2007). Wise, Darling-Hammond, McLaughlin, and Bernstein concluded that “teacher evaluation, properly done, is a difficult undertaking” (1985, p. 62). There is overwhelming evidence suggesting that teacher evaluation, as it currently stands, is ineffective (Danielson & McGreal, 2001; Darling-Hammond, 2012; Kennedy, 2010; Marzano, Frontier, & Livingston, 2011).

Research has identified several crucial influences on evaluation systems, including the provision of useful feedback, trustworthy and engaged leadership, and clearly identified purposes of evaluation, ideally formative (D. R. Davis, Ellett, & Annunziata, 2002; Frase & Streshly, 1994; Tuytens & Devos, 2017). D. R. Davis et al. (2002) argued that an outstanding teacher evaluation system is of little use if not implemented well and if the school leader is not supportive. Wahlstrom and Louis (2008) argued that school leaders are responsible for ensuring the quality of teachers' work. School leaders should be capable of providing constructive feedback to teachers. Research suggested that teacher evaluation is a complex, multi-faceted, and relationship-driven supervisory activity (Lortie, 2009; Zepeda, 2016).

Despite the evidence indicating that school leaders need to provide useful feedback to help teachers improve, other research has found that teachers frequently do not receive quality feedback from their school leaders. Weisberg et al. (2009) found that, of 12 large districts, 74 per cent of all teachers and 57 per cent of teachers in their first three years reported that principals provided no feedback or suggestions for teachers on their summative evaluations. Other studies found that the feedback principals did provide lacked specificity (Sinnema & Robinson, 2007). Finally, principals do not often link teacher evaluation to creating professional learning opportunities for teachers (Stronge & Tucker, 2003).

In recent years, teacher evaluation seems to have shifted towards formative assessment rather than summative assessment; that is, teacher evaluation is being used for teacher improvement and professional development rather than simply measuring performance. Although Frase and Streshly (1994) identified a lack of alignment between teacher evaluation and teacher professional development, more recent studies have shown a move to changing the narrative around teacher evaluation systems. Two international large-scale OECD studies, ‘Evaluating and rewarding the quality of teachers: International practices’ (Matter, 2009) and ‘Creating effective teaching and learning environments: First results from TALIS’ (Peña-López, 2009), emphasised the importance of improvement plans that identify areas in which professional development is needed. Close, Amrein-Beardsley, and Collins (2018) found that several US state departments of education have shifted their focus to encourage districts to use teacher evaluation to provide formative feedback rather than strictly for accountability purposes. In a study of teacher evaluation in a large USA urban school district, researchers found that administrators in the more effective schools tended to use evaluation data first and foremost to inform professional development, rather than to reward or discipline teachers (Reinhorn, Johnson, & Simon, 2017). These recent policy changes suggest that modifying the implementation of highly structured, state-driven, consequential evaluation policy is important for principals to develop and maintain a supervisory relationship with their teachers.

Current research suggested that successful school reform is predicated on school systems defining, identifying, developing, and rewarding educator effectiveness (Odden, 2011). A study by the MET project found that it is essential to utilise multiple methods for teacher evaluation, which should all be considered in balance, for example, value-added methods and student opinions (Kane & Steiger, 2012). Furthermore, the study also suggested that any evaluation system should take a theory-based or standards approach to building a clear and articulated framework to encourage engagement.

Evaluating the performance and impact of teachers is an integral part of ensuring the quality of teaching and to have a clear understanding of what is going on in their school, as well as how students are being affected. Any framework or model used to evaluate the qualities of teachers and teaching must prioritise


the consequential impact on students. The criterion of success for any evaluation of teachers and teaching should be framed in terms of impact on the school community and the professional growth of teachers.

The previous review, Teacher effectiveness systems, frameworks and measures: a review (Clinton et al., 2016), provided a high-level picture of the state of play of teacher evaluation and teacher evaluation measures. This review includes a series of high-level recommendations for government and policymakers relating to measures of teacher effectiveness and teacher evaluation systems. The table overleaf summarises the key features of systems for teacher evaluation that promote professional growth.


Table 6. Key Features of Evaluation Systems that Promote Teachers’ Professional Growth

Component Features

System-level policy settings and requirements

Standards must be evidence-based and support career progression The evaluation system needs to separate the formative and summative

evaluation of quality teaching There needs to be a regulatory body in place to articulate and support

evaluation and house the results Evaluative judgments must be made against standards that are progressive

across career

Quality measures of teacher effectiveness

Resources Measures need to reflect progressive development Measures must be valid, reliable, consistent over time, and unbiased Multiple methods of measurement need to be available and utilized

simultaneously Context

Measures must be adaptable to multiple contexts Remote and various communities must be able to utilize measures Multiple voices need to be taken into account, for example, student,

teachers, and parents Dimensions of measurement

Instructional practice Skill and knowledge Professionalization Teacher impact on educational communities Student learning outcomes (cognitive and non-cognitive)

Applications of measures Measurement assumptions and procedures Defensible weighting of the various components must take place in relation

to individual teachers Measures must operate across different contexts Defensible cut scores for making decisions about high quality

Conducting measurement in remote and Aboriginal and Torres Strait Islander communities

Measures must be adaptable to multiple contexts Remote and indigenous communities need to able to utilize measures Communities engaged in process and criteria for success

Evaluator expertise Training for supervisors and coaching is accessible Capacity building in the form of induction and ongoing support needs to be

available for teachers and supervisorsUse of evaluation results

Evaluative results must be utilised for educational improvement of the school community

Results need to inform policy and practice within the context being evaluated

Support decisions are required about professional growth, eg professional learning, certification at higher levels of the standards, etc.


3.1.3. Preconditions for Evaluation Systems As illustrated above, there are key requirements for building systems that promote professional growth. However, the establishment and sustainability of such systems should be supported by a series of conditions, referred to in this report as preconditions, for an evaluation system. The literature reviewed indicated that the effective evaluation systems that promote the professional growth of teachers are dependent on four categories of conditions.

Table 7. Preconditions for Evaluation Systems that Promote Professional Growth

Category Condition Application (what it looks like in practice)

Evidence

Shared purpose, explicit criteria

Clarity of teacher expectations

Common goals Connection of goals

to professional development

Useful and regular cycles of feedback

Embedded and interwoven into teaching

Collective decision making

Optimism towards future success

Alignment with standards

Collaboration Transparency of

evaluation process

Camburn & Han, 2015 Kelly, Ang, Chong & Hu,

2008 Darling-Hammond &

McLaughlin, 2011 Shen, 2015 Philips, Balan & Manko,

2014 Smylie, 2014

Relational trust

Support for teacher learning

Leadership involvement and impact on school improvement

Effective professional conversations and feedback

Co-construction of evaluation purpose

Collaboration, sharing of knowledge

Goe, Biggers & Croft, 2012 Hill & Herlihy, 2011 Tuytens & Devos, 2017 Selkrig & Keamy, 2015 Liu, Hallinger & Feng (2016) Darling-Hammond et al.,

2009

School leadership, positive school climate

Overall ‘quality of school life’

Influence of trust and motivation levels of school community members

Leadership styles that enable and reinforce professional growth and learning

Culture that embraces inquiry

De-bunking myths about repercussions of evaluation

fairness supporting teacher

agency Supporting teachers

in analysing and reflecting on their practice

Time allocated for evaluation practice

Teacher well-being is valued

Motivate and support learning of teachers

Teacher concerns are shared and listened to

Robinson, Hohepa & Lloyd, 2009

DiPaola & Hoy, 2014 Patton, Parker & Pratt, 2013 Anderson et al., 2012 Hill & Herlihy, 2011 Timperley, 2008 Bruneau & Saxe, 2012 Hargreaves & Fullan, 2012

Evaluative Positive mindset Feedback loop Le Fevre, 2015


Category Condition Application (what it looks like in practice)

Evidence

mindset for evaluation Capacity building

Active engagement and participation

It should be noted that the preconditions identified do not operate in isolation and have been found to extensively affect each other. For instance, relational trust between teachers and leaders supports the development of a positive school climate.

Shared clarity of purpose and explicit criteria Research indicated that shared clarity of purpose and explicit criteria are essential when teachers explicitly reflect on activities to affect change in professional practice (Camburn & Han, 2015; Kelly, Ang, Chong, & Hu, 2008). The Development Framework (ATPDF) describes components needed to improve teaching practice: clarity of teacher expectations, supported by frequent, useful feedback within a regular performance and development cycle with access to high-quality teaching and learning support. Consequently, performance and development processes and teacher goals should reflect the overall approach to teaching and learning within a school and should be consistent with the school plans. It is important that school plans and school-wide approaches to professional learning are aligned, and that teachers and school leaders experience performance and development as something interwoven with the various activities in which they are engaged, rather than as separate and additional processes. Darling-Hammond and McLaughlin (2011) also suggested that it is important for teachers to participate in collective decision-making pertaining to the ‘what’ and ‘how’ of their learning in order to increase a sense of ownership over their learning and potential for affecting professional change. However, it can be argued that the foundation of the standards described requires not only the shared clarity of purpose and explicit criteria for evaluation, but also depend on the extent of relational trust and positive culture existing within the school in order to facilitate effective professional conversations.

Relational trustRelational trust refers to the trusting relationship between two or more parties that allows change to take place; typically, this refers to the trust between school leaders and teachers that allows school and teaching improvement to take place (Darling-Hammond & Richardson, 2009; Hallinger & Lu, 2014; Keamy, 2016; Le Fevre & Robinson, 2015). Relational trust has been identified as one enabling factor that mediates the efforts of school leaders to motivate and support the learning of teachers (Cosner, 2009; Hallinger, Heck, & Murphy, 2014; Li, Hallinger, & Walker, 2016). Teachers are positively disposed to “de-privatising” practices, in which teachers collaborate and participate in various evaluation processes. These practices include peer classroom observations or collaborative feedback sessions in which teachers discuss with mentors and school leadership about their practice and identify areas for growth and learning. Darling-Hammond and Richardson (2009) advocated for the fostering of strong relationships amongst teachers and school leadership to facilitate professional learning and development through effective professional conversations involving processes of feedback and collaboration (Goe, Biggers, & Croft, 2012; Hill & Herlihy, 2011; Selkrig & Keamy, 2015; Tuytens & Devos, 2017). The Development Framework (ATPDF) positions formal and informal feedback as essential to improving teacher practice, as this informs growth and access to high-quality professional learning. Ritter and Barnett (2016) argued that rigorous evaluations are far more likely to be welcomed in an environment of trust and shared responsibility, which is related to supportive school leadership and positive school climate.

Supportive school leadership and a positive school climateSchool climate refers broadly to the ‘quality and character of school life’, and in the literature tends to be conceptualised as comprising four domains: safety, teaching and learning, relationships, and environmental structure (Cohen, McCabe, Michelli, & Pickeral, 2009). Positive school climates have been shown to promote trusting relationships, advocate for “protected time” to complete evaluations, and provide


available resources to support evaluation, all of which impact the success of evaluation implementation. Opfer, Pedder, and Lavicza (2011) remarked that a school’s organisation may enable or constrain teachers. As such, the establishment of a positive school culture should be considered fundamental to the implementation of an evaluation.

Existing research has argued for appropriate and sufficient guidance, support, and leadership within the school environment and administration to support the professional agency of teachers (DiPaola, 2014; Patton, Parker, & Pratt, 2013; Patton, Parker, & Tannehill, 2015). Within the APST, under the domain of professional engagement, teacher professional agency is encouraged with teachers tasked to identify their own learning needs and analyse, evaluate, and expand their professional learning, both collegially and individually. Teachers need to be supported by their principals and other instructional leaders in analysing and reflecting on their own practice and learning from feedback.

The role of principals has been examined in relation to impact on the overall school culture and model of professional learning adopted by the school (see Derrington & Campbell, 2015; Liu, Hallinger, & Feng, 2016). Tuytens and Devos (2017) have suggested that the school leader has an important role to play for both teacher evaluation and the broader school organisation. As it is not simple for school leaders to learn the skill of supporting teachers in the process of evaluative feedback for professional learning, Robinson et al. (2009) suggested the school leader engage in ‘open-to-learning’ conversations within the contexts of teacher evaluation. They suggest school leaders should be able to disclose their views, listen to those of others, and remain open for mutual influence. Hallinger (2011) called for more research on how leadership strategies influence different settings and impact on teacher evaluation. The Australian Professional Standard for Principals (APSP) describes standards for leading improvements and change, where principals are responsible for setting the culture of the school; the implementation of clear, evidence-based plans and improvements; and setting the overall direction of the school. In addition, principals also need to advocate for a positive culture of learning and support, with high expectations set through collaborative planning, monitoring, and reviewing of learning. The role of principals is to support others to build capacity and treat people fairly and with respect. All these elements are central to a performance and development culture. While it is acknowledged that principals have a central role to play in evaluation implementation, a performance and development culture cannot be driven by one person alone. Leadership must come from all levels, from those with and without formal leadership positions.

Evaluative mindsetClinton and Dawson (2018) have argued that the notion of evaluative thinking is essential as it supports progression that is essential for change. Evaluative thinking involves individuals seeking evidence about their impact beyond the reflection of their practice, embodying the notion of assessment and judgement in tandem with, or in addition to, consideration and thought about their practice. Essentially, the suggestion of a practitioner change in mindset for the purposes of capacity building and enhancement of effective teaching practices incorporates the inclination to engage in a feedback loop through active engagement in the process of evaluation. Hattie (2008) has also shown that it is what teachers know, do, and care about that is the most crucial for student learning. Similarly, Le Fevre, Robinson, and Sinnema (2015) suggested that it is important for teachers to have a positive mindset about the purposes of evaluation in order for effective professional conversations to take place.

However, existing research has also acknowledged that changing the culture of evaluation in education is difficult (Jensen, 2010) and that many teachers view evaluation as a meaningless exercise to be endured (Danielson, 2011; Weingarten, 2010). For example, in an evaluation of the implementation of the APST (Clinton et al., 2015), the performance appraisal was often the cause for great concern to teachers. Many educators participating in the evaluation were concerned that the APST can be used as a compliance measure, rather than support career progression. Therefore, teacher evaluation needs to commence with


the teacher so that the information becomes an element of a developmental process of evaluation, whereas evaluative thinking needs to be at the heart of this process.

The preconditions also reflect a consistent message evident in the ATPDF, which is that there are certain factors that must be present at the school level to support teacher learning to flourish; there should be a focus on student outcomes, a clear vision of effective teaching (as drawn from the APST), a strong role of leadership in creating a culture and adequate flexibility to respond to different contexts and starting points, and coherence between teacher goals, school plans, and school-wide approaches (AITSL, 2018b, p. 5). Reviewed literature demonstrated a large degree of overlap with the preconditions and the factors stated in the ATPDF.

3.1.4. Principles for Selecting Evaluation Resources Once a system is in place for teacher evaluation that promotes professional growth, planning for evaluation practices (along with determining necessary infrastructures) is required. Through the literature scan, principles for guiding practices and the selection of evaluation resources have been identified, on the assumption that there is already a system in place with the aforementioned preconditions. It should be noted that the term ‘resource’ is defined in a broad sense and may include practices, tools and human, financial and physical facilities.

As stated above, any principle developed to guide planning for teacher evaluation must be based upon an evidence-informed understanding of teaching. Teaching is complex, and fit-for-purpose use of assessments tools to foster teacher effectiveness and professional development are needed to capture that complexity. However, this needs to be balanced with practical considerations, such as cost-effectiveness, which is not solely about purchasing a tool but also about infrastructure and administrative requirements in the use of a tool or evaluation practice(s) (Tri-Partite Alliance, 2014). Kennedy (2010) argued that there is no single, overarching theory of teaching or of teacher learning, and it is of value to examine what teachers are doing and how we can help them improve. The plethora of tools and professional development initiatives rely on the identification of a central problem of practice and aim to inform methods or ways in which teachers can improve their practice. Thus, in addition to consider cost-effectiveness, focusing on a relevant goal for improving practice is an effective way to ensure that assessment and evaluation practices generate actionable feedback to focus a teachers’ professional development activities and decisions (Darling-Hammond, Hyler, & Gardner, 2017; Filipe, Ferreira, & Santos, 2015; Kennedy, 2010).

It should be noted that the principles were also informed by the previous review of teacher evaluation systems, measures, and frameworks (Clinton et al., 2016). Relevant findings from this review indicated that effective teacher evaluation measures include opportunities for teachers to engage in self-reflection through personalised assessments, use of multiple methods of data and information gathering, for example, peer interviews, classroom observations, peer or student ratings, or evaluations by school leaders. Teaching is an activity that draws on multiple sources of knowledge by integrating different knowledge types, context-relevant practices, relationship building and commitment to equity and care. Best practices for these professions include a common language (Hattie, 2008) and shared frames of reference for contexts and practices.

In education, the overlap between the roles of the assessed (eg teacher) and the assessor (eg school leaders/principals) have been identified in the evidence-based and collaborative principles, along with understandability and professional growth. These principles prioritise clear and cyclical feedback to further support effective teaching practices and professional growth, where the onus is on individual teachers to drive their own learning and development (Darling-Hammond & McLaughlin, 1995). Parallels are evident between teaching and clinical and allied health, where there is an emphasis on the practitioner to drive their own assessments based on the competencies being measured, ensuring the choice of assessment tool accurately matches the selected competency (Department of Health and Human Services, 2019). Materials on the Victorian state government website for allied health, for example, indicate that practitioners are


supported to undertake a self-driven process of assessment via the provision of structured materials to assist in identifying the scope of practice, competency, and capability. The suggested starting point is the use of self-assessment tools against the practice and assessment framework, allowing practitioners to assess their current position, and then locate the relevant section of the framework or resource kit to address their identified need (Department of Human Services, 2012).

The process of selecting individual tools and strategies should be based on contexts, purposes, opportunities for data collection, and the capacity and support of supervisors. This was reiterated throughout the reviewed documents and assessment literature (Darling-Hammond et al., 2017). For example, clinical supervision and performance reviews of allied health practitioners are also regarded as a fundamental part of ensuring an effective workforce that is capable of providing safe and high-quality care, in addition to utilising self-assessment tools to identify areas for targeted action. Similarly, the Australian Association for Social Workers iterates that its members are responsible for their own continued professional development. Individual members are encouraged to plan and seek out their own professional development opportunities online or through targeted events (Australian Association of Social Workers, 2019).

Continual professional growth, while indisputably important across the human service sectors included in the review, is also dependent on the regularity of assessments over time, as it requires the accumulation of data or evidence about performance over a period of time, in order to provide opportunities for feedback and improvement of practice (Nursing and Midwifery Board of Australia, 2019; Tri-Partite Alliance, 2014).

Reviewed literature in education highlighted the importance of a collaborative approach to both teaching and self-evaluation, which would necessarily involve considering not only the views of the assessed, but also views of other stakeholders such as students (Ofsted, 2015). Student-based feedback, which may involve students in the feedback or evaluation process and the measurement of teacher practice, therefore becomes an important consideration. According to Fleming, the examination of student’s voice incorporated in the assessment of teachers are hallmarks of assessment that supports effective teaching practices and encompass a range of terms and activities that target a “repositioning of students to facilitate their engagement with their teachers and schools” (2017, p. 233). There is a significant focus on the contribution of student voices at both classroom and system levels, with research postulating student transformation through engagement and active participation in their learning (Charteris & Smardon, 2019).

Comprehensiveness and relevance are deemed essential to evaluate the spectrum of competencies required for achieving, sustaining or progressing in professional competence across human service sectors. Evaluation needs to capture relevant data that could add value to personal and/or group reflection on competence (Crossley & Jolly, 2012; Govaerts & van der Vleuten, 2013; Holmboe, 2017). For example, in the nursing and allied health sectors, there is a strong emphasis placed on the comprehensiveness of the assessments undertaken to ascertain competency in the assessed skill, including the use of multiple sources of data, modalities, samples, and different assessors to capture evidence of competency (National Education Framework Cancer Nursing, 2019). However, there is also a recognition that this should be determined in alignment with the purpose of the assessment, whether it is intended to be formative or summative in scope, or a combination of both. Features of teacher assessments have also focused on the accuracy and validity of multiple modes of evidence collected within specific contexts in alignment with overarching policy and guidelines (AITSL, 2017).

Within the allied health sector in Victoria, a credentialing, competency, and capability framework in line with a standardised and consistent approach to evaluation has been adopted (Department of Human Services, 2012). Resource toolkits have been provided to developers in the planning and development of appropriate assessment tools in the measurement of capabilities and competencies as allied health practitioners deliver services across a wide range of sectors, contexts, and settings. The allied health framework utilises clear governing principles of practice when guiding the assessments, including aspects of


best practice, based on evidence drawn from local, national, and international examples. The aim is to grow the capabilities of practitioners, create and share learning cultures, develop new roles, and advance and extend scopes or practices following assessment (Department of Health and Human Services, 2019). In short, the evaluation system aims to support professional growth.

1) STANDARDS FOR ASSESSMENT AND EDUCATIONAL EVALUATIONWithin the education space, it is essential to understand the concept of assessment for learning, in particular, how this relates to assessment for the evaluation of teaching and teachers. When considering the development of principles for the selection of resources and evaluation planning, we must consider key standards of assessment to ensure that all tools and resources are not only fair but are also implemented in an appropriate manner to encourage development. The literature on the principles of assessment provides a useful foundation for this. Quality assessment must include valid interpretations based on reliable measurement tools, that have consequential impacts on the decisions, action, and next steps in the learning cycle (Brookhart, 2011).

This places much emphasis on providing defensible and meaningful score reports that are both correctly interpreted and lead to consequential decisions. Such assessments aim to provide feedback to the test administrator and the test user about past (more summative) and future (more formative) decisions. They also lead to evaluations about the impact on the experiences of the learner and the learning experience (including the assessment design). Developing meaningful score reports requires assessments that have appropriate psychometric properties (eg, reliability, validity, lack of bias, reproducible).

The Australian Council for Educational Research, in the document Principles of Good Practice in Learning Assessment, suggests that there are several key quality concepts in building assessment for learning (ACER, 2017):

Fitness for purpose Clarity and consistency of purpose Objectivity and independence Transparency and accountability Technical rigour Ethicality and fairness.

It is suggested that good practice in learning assessment ensures that key quality concepts are met at all stages of an assessment program. The document goes on to describe best practice in implementing any assessment for learning to ensure that the quality concepts are met. The figure below shows the 14 key areas of a robust assessment program, which range from identifying and defining policy goals and education issues, to designing and implementing the assessment, through to analysing and reporting the data and informing the initially defined policy goals and issues.


Figure 8. The 14 key areas of a robust assessment program (ACER, 2017).

To be robust and effective, it is important that any assessment uses well-founded methodologies in each of the key areas of implementation. It is our contention that any assessment or resource utilised in evaluations of and with teachers should also be upheld to the same level of standard as any assessment utilised in a school setting. This is the foundation for the development of the principles for guiding the selection of teacher evaluation tool or resource.

In addition to the standards for assessment, it is also important to ensure that any teacher evaluation process adheres to the standards for educational evaluation. The Joint Committee Program on Educational Evaluation defined standards (a total of 30) that describe what effective educational evaluation looks like. These are grouped into four categories (Yarbrough, Shula, Hopson, & Caruthers, 2010):

1. Utility: all evaluations should meet and address the information needs of the end-user audiences. 2. Feasibility: all evaluations should be realistic, efficient, diplomatic, and prudent.3. Propriety: all evaluations should be conducted in accordance with relevant legal, ethical standards, and

codes of practice, with the utmost regard for the welfare for all involved in the evaluation, as well as those who will or could be affected by the results.

4. Accuracy: all evaluations should gather and utilise technically adequate information about the components that determine the worth or merit of the evaluand.

The establishment of an evaluation system that supports the professional grown of teachers requires adherence to these standards. The Joint Committee Program standards have also been widely tested and applied in other sectors, including public health, and are endorsed by the American Evaluation Association and the Australian Evaluation Society (Yarbrough et al., 2010).

2) DRAFT PRINCIPLES FOR EVALUATION RESOURCESDrawing the findings of the literature scan together, and incorporating relevant standards for educational assessment and evaluation, 11 draft overarching principles with descriptors were developed. The principles (detailed overleaf) characterise the values that tools, resources, and evaluation practices should embody if they are being applied with the intention of promoting the professional growth of teachers.


Table 8. Principles for Teacher Assessment and Evaluation Derived from the Literature Scan

Principle Description

Authentic Authenticity can be achieved when assessment, evaluation or feedback occurs within the context or setting of the practice of the person being evaluated to capture accurate evidence of skills, attitudes and knowledge. Evidence gathered needs to reflect real-life practice and needs to be collected in a manner that ensures honesty and validity.

Evidence-based Assessment, evaluation or feedback should be carried out by educators who have comprehensive knowledge of and relevant expertise in the skills to be assessed, the expected standards and performance assessment. Evaluations must be justified based on evidence contained in the assessment used and evidence presented must provide proof of competence when matched against the relevant standard(s).

Collaborative Assessment, evaluation or feedback should involve high levels of communication, reflection and reinterpretation of performance. May include collaboration between the assessor or school leader and the practitioner to enable validation of judgements made or may involve collaboration between peers.

Context-specific Assessment, evaluation or feedback should be tailored to practice settings, individual school priorities and school context.

Cost effective Assessment, evaluation or feedback should consider factors such as infrastructure requirements, administrative support, analytical expertise, training requirements for assessors (where relevant) in addition to the direct costs of using a specific strategy or tool.

Comprehensive Comprehensive assessment, evaluation or feedback may use multiple modalities, multiple samples, or multiple assessors. Evidence can be direct (witnessed by the assessor), indirect (eg reviewed later), supplementary (eg third-party reporting) or involve self-assessment. Together such methods provide the evidence needed for the judgement of competence or performance against the standard(s) to be made.

Understandability Assessment, evaluation or feedback domains, purpose, standards, data and feedback must be clear and easy to understand for all parties.

Relevant Assessment, evaluation or feedback should be linked to a diverse set of profession-specific standards, competencies or performance metrics and must be relevant to the context.

Professional agency Assessment, evaluation or feedback data may determine and prioritise areas for professional growth, identify professional learning opportunities and allow planning for career development via the provision of actionable feedback.

Regular Assessment, evaluation or feedback should be regular, which may involve accumulating data or evidence about performance over a period of time. Regular assessments may provide ongoing progress monitoring and yield feedback on a variety of specific practices.

Student-based Assessment, evaluation or feedback should be designed to gauge and measure student growth, engagement, interaction, enjoyment or understanding. These might involve students in the feedback/evaluation process, measurement of growth or entail observation of classroom interactions.

Notably, there are overlaps between several principles. For example, authenticity, context specificity, comprehensiveness, and relevance in assessments have been derived from a strong emphasis in reviewed sources that assessment should be undertaken within the context of professional practice. The reason for this is to ensure that evidence of skills, attitudes, and knowledge is captured accurately (National Education Framework Cancer Nursing, 2019). If this occurs, the authenticity of the assessment and the information generated is more likely to be assured (National Education Framework Cancer Nursing, 2019; Tri-Partite Alliance, 2014). Further, the focus on ‘in-situ’ assessment also ensures that the assessment, and more


importantly, the evaluation practice and interpretation of information gathered, occurs in the environment in which the professional works. This helps ensure that the information gathered can be acted upon, and that it is more likely to be accurate and interpreted with a degree of validity (Tri-Partite Alliance, 2014).

The draft principles and descriptors were tested through stakeholder interviews and group consultations prior to the Delphi surveys and the stakeholder workshops in Sydney, Melbourne, Darwin, and Perth.

3.2. Stakeholder InterviewsAs described, a number of consultations were undertaken with stakeholders from the education, justice, higher education, and health sectors. Stakeholders were asked to provide their perspectives on the current status of teacher evaluation systems, their worth, and potential hinderers. The participants were asked to focus on the key characteristics of the quality of evaluation systems in their profession, as well as the main reason for implementing an evaluation system.

Table 9. Individual and Small Group Stakeholder Consultations

Stakeholder Group

Dr Ken Muir (Department of Education, Scotland) Scottish education system

Professor Michael Fullan Expert

Dr Ray Smith Expert

Professor John Hattie Expert

Dr Julie Smith Expert

Members of the General Teaching Council for Scotland Scottish education system

Primary school teachers (n=3) Teachers

Initial teacher educator Expert

Dr Duncan Howard (MCCC4 GP Training) Health

Justin Goldsmith (Victoria Police) Justice

The notes from these interviews were thematically analysed and a number of key themes emerged. These themes are presented in the table overleaf. A measure of the depth of presence of the theme was estimated. The absolute values provided for the depth of presence should be taken more as an indicator of saturation rather than a judgement.

4 MCCC: Murray City Country Coast GP Training, Victoria.


Table 10. Themes from Individual and Small Group Interviews

Theme Description Consistency and depth of theme Sc

ottish

ed

ucati

on

Expe

rts

Teac

hers

Justi

ce

Heal

th

Clarity of purpose

The transparency of process as well as an understand of what is at stake or what actions may occur as a consequence of the process

100% ✔ ✔ ✔ ✔ ✔

Evaluative mindset

Development of the desire to understand the effectiveness and efficiency of practice and what steps to take to progress further

70% ✔ ✔ ✔

Outcome Ensure fit with student and teacher outcomes 75% ✔ ✔ ✔ ✔

Resources and Policies

Access to assessments, frameworks and documents to support the process

100% ✔ ✔ ✔ ✔ ✔

Appropriate setting/ conditions

A culture to support the process 75% ✔ ✔ ✔ ✔

Leadership support -Modelling the process

School leaders engage in the process and model behaviour of self-reflection

65% ✔ ✔ ✔

Whole of organisation process

Everyone on the organisation is engaged and process distributed

75% ✔ ✔ ✔ ✔

Growth/ professional learning provision

Professional learning must be targeted to process 75% ✔ ✔ ✔ ✔

Link to professional goals

Relationship to an individual’s professional goals 100% ✔ ✔ ✔ ✔ ✔

Feedback Clear and useable feedback for all participants 100% ✔ ✔ ✔ ✔ ✔

Build evaluation capacity

Workforce capacity of all who engage in the evaluative process

50% ✔ ✔

Time Dedicated time to be engaged 100% ✔ ✔ ✔ ✔ ✔

Infrastructure A system of collated data and reporting 75% ✔ ✔ ✔ ✔

System support The sector provides support and guidance for the process 70% ✔ ✔ ✔

Implementation

Process is enacted 90% ✔ ✔ ✔ ✔

These themes can be collapsed into several categories: the purpose, necessary context and conditions, process and implementation of evaluation, and final outcomes. Each of these is described in more detail below.


PurposeAll participants suggested that clarity of purpose for the evaluation system was key. Furthermore, the characteristics of the evaluation process, assessment or feedback must be designed to support teacher professional practice growth. Feedback, which was noted as critical, must also be focussed on supporting targeted professional practice growth.

It must always be about continued professional growth. (Prof M. Fullan)

Teacher evaluation systems must be designed to improve teachers' instructional capability so that they can do her or his most effective job that all students achieve a year's worth of progress for a year's worth of input. (Dr Ray Smith)

The Professional Development Assessment is aimed at the continual development and support of individuals. It has been used for approximately 20 years and continues to be improved. It is not designed to be punitive. (Justice)

Context and conditions necessary to implementIt was considered important to understand the context in which people are working, and in particular, any changes they may face in the evaluation process. One area of significance in the education sector was the use of systems to support early career teachers through the induction process. In Scotland, this was a key strategy; teachers could be supported through induction by acknowledging their current career stage and the context in which they were teaching. In addition, all participants talked about developing appropriate conditions so that evaluation systems could add value. Phrases included, but were not limited to: trust, relationships, and strong rapport.

In most contexts, teacher evaluation is about the system that the teachers are in. Schools want teachers doing well and having an impact but have a lot of constraints for teachers within the process. They are thinking at a systems level and not looking at the teacher as an individual. Because of this, teachers are feeling that it [evaluation] is being done to them. (Primary school teacher 1)

Being in a context in which there are good, trusting relationships – particularly the relationship between the registrar and their medical educator/supervising GP [is important]. (Health)

Process and implementationRegardless of the sector, the idea of using appropriate processes to implement evaluation was seen as a constant variable. Degree of implementation is an issue as well as fidelity. It was acknowledged that there are varying quality and degrees of implementation of evaluation systems. The quotes from the teachers demonstrate the differences across the systems. Invariably, when asked about the system, most people suggested that there was difference between the formal performance review process and internal support of coaching.

Feedback was also a critical part of the process. There was much discussion about the way feedback is given and its connection to growth.

We are assigned a coach for the year who we connect with throughout the year to go through template of appraisal. I find it helpful to talk through the process of what I want to work on and make it meaningful. However, I feel that having more of a system figure out direct impact without having to do so much extra. The coaches we have are great practitioners but are not great evaluators. (Primary school teacher 1)

[Evaluation is] Not individualised for each teacher. Becomes a ‘chore’ and something to just get done, rather than using it as reflecting and evaluating time. Observations are too formal and not always an accurate show of how a teacher runs lessons day to day. (Primary school teacher 2)

It needs to be doable and easy, because sometimes the performance review is a waste of time. (Primary school teacher 3)

Capacity to evaluate was different and considered a part of the evaluation process. It was acknowledged that this is an issue for teachers and supervisors. This was also true of the professions more generally.


Teachers are not evaluators. We get taught how to teach and how to evaluate learners, but we need to be able to feel comfortable putting that critical lens on ourselves. We have been taught that if we feel that we are struggling and share it with the wider school, such as leaders, coaches and mentors we have a lot of extra work. The system then labels that teacher as a concern and trust goes out the window. (Primary school teacher 3)

Culture and mindset The culture of the profession was also seen as a major concern. The statement from stakeholders from the Scottish education system was indicative of the process that has been established. It was suggested that, consequently, there is an evaluative culture, which provides a highly supportive context for teacher evaluation. This in turn supports professional growth to occur.

The school culture is a big part of the evaluation and the teachers’ mindsets are the biggest barriers. Culture + teacher mindset = huge barriers to development. (Primary school teacher 1)

Supporting teachers as learners is a part of the DNA of the Scottish system. (Focus group, GTSC)

The Professional Development Assessment is linked to pay, so it has its inbuilt incentivisation, though some members might find it a bit of a burden. (Justice)

Professional learning is a right not a privilege. (Ken Muir)

The Professional Development Assessment is regarded as a genuine investment in members’ careers. Good managers have learnt to utilise it to maximum benefit. It has become a ‘business as usual’ practice, realising that at least 70 per cent of a member’s professional development happens in the workplace. (Justice)

Other emerging themesAdditional views from interviewees included consideration of third-party input, where people other than school staff or the assessor, for example a parent or caregiver, may have something useful to contribute about the progress of the teacher being evaluated. In the case of general practice clinic, a practice receptionist is a key knowledge holder who may have observations or feedback to contribute. Dr Julie Smith issued a final caution:

…operating on a belief that all teachers must be evaluated every year, using a single-evidence source (eg, administrative observation), based on a small sample of teacher behaviours, using an observational instrument by a single administrator that attempts to capture more data points than is realistically possible within a typical classroom period, at the same time ignoring one of the most reliable data sources--student voice, and not incorporating teacher self-assessment is a key problem [to avoid]…

3.3. Peak Body and Group ConsultationsSimultaneous to the stakeholder interviews, two peak body and group consultations were conducted for the purposes of refining the principles developed from the literature scan before testing in the stakeholder workshops and the Delphi survey. It should be noted that these consultations were identified as opportunities in collaboration with AITSL. The School Leadership and Teaching Expert Standing Committee (SLTESC) took place on 3rd April 2019, and the Australian Alliance of Associations in Education (AAAE) took place on 1st June 2019.

At both consultations, participants were introduced to the study and asked to review the principles and rank their importance in relation to one another. The graphs below illustrate the ratings for the least and most important principles.

The figure below indicates that most participants at AAAE ranked context specificity as most important, followed by collaboration. For SLTESC participants, student-based was also very important along with collaboration. It is interesting to note that there was a great deal on variance between the bodies in relation to the importance of the principle of context-specific.


Regular

Understandable

Evidence-based

Comprehensive

Context-specific

Authentic

Relevant

Collaborative

Student-based

8%

8%

8%

31%

23%

23%

6%

6%

6%

18%

18%

24%

24%

AAAESLTSEC

Figure 9. Stakeholder group consultations ranked most important principles.

Rankings for the least important principles comparatively indicated more consistency between the groups in rankings, particularly for cost-effectiveness; however, there were some differences across items, eg whether a resource should be student-based, or whether resources should be made available for regular assessment.

Evidence-based

Regular

Student-based

Understandable

Comprehensive

Cost effective

0%

6%

6%

6%

18%

65%

8%

15%

15%

62%

AAAESLTSEC

Figure 10. Stakeholder group consultations ranked least important principles.

The results from these consultations assisted with the refinement of the principles for subsequent use in the national workshops and the Delphi survey. The findings from each of these large data activities are discussed in the following sections.


3.4. Stakeholder WorkshopsThe findings from the four national stakeholder workshops convened in Sydney, Darwin, Melbourne and Perth are discussed in this section. As described in Section 2.2, there were multiple aims of the stakeholder workshops, including testing the principles developed in the literature scan. A total of 47 education stakeholders attended the workshops.

The activities undertaken in this workshop were conducted to glean practitioner perspectives on the elements key to building the Framework, that is, to guide the selection and use of evaluation tools that aim to support the professional growth of teachers (see the table below). The organisations represented at the workshops are included in Appendix B.

Table 11. Stakeholder Workshop Activities Overview

Activity Intended purpose and aims of activity

1. Understanding perspectives of effectiveness and professional growth

To understand users’ perspectives of teacher effectiveness and professional growth, including similarities and differences in views, and to frame subsequent discussions and activities.

2. Sharing and understanding from the sector

To share with workshop attendees the principles developed from the literature scan and findings from ranked item exercises conducted during previous consultations and modified Delphi survey responses.

To gather reactions to the emerging findings and the degree to which these findings vary or align with their understanding of the key principles that should underpin framings of teacher assessments to promote professional growth.

To discover guiding principles of how such assessments should be conducted.

3. Professional growth scenarios To identify the resources/supports that would be necessary for teachers and school leaders to conduct assessment of practice and monitor progress towards goals.

To determine important context-specific and practical considerations for conducting these assessments.

4. Planning for evaluation, selecting teacher assessment tools or methods of practice

To assess the key elements of a range of prospective and different evaluation resources that could be applied from their perspective (teacher, school leader).

To determine the steps for selecting and operationalising a resource based on the specific context.

To determine how attendees make decisions about appropriateness to suit the purpose of assessment, use of information generated by the tool to fulfil the identified purpose, and the use of information and supports needed to firstly identify the assessment and to use the selected resource.

Three activities were undertaken by the participants in all four workshops. Findings from conducting these activities at the stakeholder workshop are provided in the following sections.


3.4.1. Activity 1: Defining Teacher Effectiveness and Professional GrowthWorkshop participants were asked to individually define what ‘teacher effectiveness’ and ‘professional growth’ meant to them. They then shared their definitions with other workshop participants to identify similarities and differences. The table below details the common themes across the definitions with an indication of consistency in the presence of each theme across the workshops.

Table 12. Definitions of Teacher Effectiveness and Professional Growth

Theme Workshop 1 Workshop 2 Workshop 3 Workshop 4

Enhance/increase student outcomes ✔ ✔ ✔ ✔

Promote teacher reflexivity & responsivity

✔

Self-awareness and drive to improve ✔ ✔

Lifelong learning mindset ✔

As suggested above, definitions of teacher effectiveness were consistently associated with achieving or supporting student learning outcomes.

…the ability to increase students’ knowledge/skills/attitudes in a defined way. (Workshop 1 participant)

…capacity to enhance the outcomes of students related to efficient practice. (Workshop 1 participant)

Some definitions of teacher effectiveness also included practices to promote teacher reflexivity. Responsivity in meeting diverse socio-emotional needs and academic growth were also defined as aspects contributing to teacher effectiveness, which was seen:

…as a complex combination of increasing student skills, knowledge & attitudes within a rich & diverse learning environment stemming from a robust teacher knowledge base. (Workshop 1 participant)

Another theme was related to the self-awareness and drive of teachers to be able to effect improvements in practice through reflective processes:

Knowing your impact as a teacher and working to continually better outcomes. A high level of self-awareness of context and where you’re in. Knowing where you need to be better. (Workshop 3 participant)

… understanding what is working to promote student engagement and learning outcomes. It is also about being able to reflect on practice in this way as well, to recognise what is not working and to adapt practice accordingly. (Workshop 2 participant)

Workshop participants were also invited to share their views on the aspects that constitute teacher professional growth. Like teacher effectiveness, most definitions were associated with supporting student learning outcomes:

Professional growth means equipping oneself with knowledge and skills to support students. (Workshop 1 participant)

Improved practice related to self-fulfilment and student outcomes. (Workshop 1 participant)

Other responses provided in relation to professional growth indicated that it constituted a mindset of lifelong learning and pursuit of professional development to meet their needs and achieve their professional goals:

Continuous learning; no ceiling; classroom and beyond; combination of individual’s interest/specialty and the broader role and responsibilities of being a teacher; well-rounded professional; balance between skill area and growth of your whole professional capacity; teacher-driven pursuit of professional development; informed by teacher’s needs. (Workshop 4 participant)


Professional growth requires a person to be driven and adaptable to reflect on their current knowledge, skills and effectiveness, and then understand how to progress these further to achieve their professional goals. (Workshop 4 participant)

This activity provided framing for the subsequent group activities in considering how and what evaluation, feedback or assessment could support different teachers’ professional growth scenarios. These scenarios were provided to participants in the form of short vignettes.

3.4.2. Activity 2: Professional Growth ScenariosThe purpose of the second activity was to ascertain stakeholders’ views of how, in different scenarios, a teachers’ professional growth needs could be identified and assessed.

The comments from participants, which are provided in the following table, indicate that most participants highlighted the role of mentoring as an avenue to identify needs. This was common across the variety of vignettes shared at the workshops which depicted teachers in metropolitan and remote school settings, both in primary, secondary, and special schools, and school environments in which varied degrees of supportive learning culture were portrayed.

Table 13. Perspectives from Teacher Professional Growth Vignettes

Approaches to identify needs

Example comments Workshops providing a related comment (n)

Workshop location

Self-awareness/ Teacher Characteristics

“Need to change perception towards learners first. Build on mentorship. Grow this.”

“Misplaced confidence – preventing self-reflection”

“Be able to express their own vulnerability [and gaps in practice]. [The teacher] needs to see themselves as an on-going learner.”

“Bring [the teacher as a] learner’s needs to the forefront (instead of PD points).”

2 SydneyMelbourne

Supportive leader “Provide a challenge – build on strengths – book collaborative planning with leadership team.”

“School culture – leadership not leading teacher development”

4 SydneyDarwinMelbournePerth

Mentoring/Leadership “[the teacher in the scenario] Requires a thoughtful, strong (competent) leader to generate/facilitate professional development. Mentoring leadership; perception of professional learning and development; Reduce compliance focus”

“teacher mentoring/goal setting/mentoring.”

“May require a coach/critical friend within a structure.”

“Need for a ‘right mentor’, and if such a person could not be found in the same school, then this person could be a person external to the school, particularly in smaller schools”.

4 SydneyDarwinMelbournePerth


In addition to the common comments about mentoring, there was an explicit focus on the importance of meeting the learning needs of students and ensuring that school leaders are supporting teachers’ professional growth through formal or informal collaborative discussions and the co-construction of learning plans. This speaks to a collegial process and trust that is necessary, and that is explicit in the selection of any tool or evaluation process undertaken. Additionally, comments arising from group discussions tended to suggest the importance of the ‘teacher as a learner’ across the varied scenarios: that is, a teacher who identifies gaps in their practice and adopts an appropriate mindset for continual professional growth.

3.4.3. Activity 3: Steps to Select and Use a Tool for Assessing Professional Growth

The third and final activity of the stakeholder workshops involved the participants continuing to use the professional growth vignettes and reviewing a selection of common teacher assessment tools and evaluation resources and identifying the steps they would take to select a tool or resource for use in the scenario described in each vignette. A Think Aloud technique was used in this workshop activity to ensure participants had the opportunity to share their thinking processes in pairs.

Participants were given summaries describing the key features of a range of different evaluation resources (including the key purpose, what it measures, and how it works) and were asked to adopt either the perspective of a teacher or a school leader. Individual participants were asked to write down the exact, operational steps they would undertake to select an assessment tool or resources, including any questions or considerations that factored into their selection process.

These steps and reflections were then shared in paired discussions, to check for commonalities and differences in the steps involved with the selection of tool. Data for this activity were transcribed from the structured workshop sheets, and an inductive thematic analysis was conducted to develop themes from the data. These themes were then organised according to the stages of the cyclic evaluation process described later and detailed in Part B of the report.

All four workshops groups identified as an initial step the need to ‘consider context’. Groups noted the need to consider the teacher and school context, often through group discussions with peers and school leadership.

Considered the school context and situation – resourcing, students, etc. (Workshop 1)

Talk with the teacher’s direct manager about their perceptions of her as a teacher (Build my understanding). (Workshop 3)

Groups also identified the need to ‘investigate tools’ during initial steps. Mostly, responses that were coded into this theme tended to describe steps that involved searching for tools and checking their availability.

I would then explore the tools above to gain Zara’s perspective. This may help her refine/reflect. (Workshop 4)

A third theme commonly noted in the stakeholder workshops was ‘determine tool applicability’. Steps identified in this theme tended to relate to determining the extent to which the tool met teachers’ needs, particularly their professional development needs.

Will this tool help identify practical implications/recommendations? (Workshop 3)

Now need a tool that supports [the teacher] to self-assess or extend upon [their] current knowledge/mentoring skills. (Workshop 1)

In the later steps, groups tended to describe actions that were related to ‘setting up implementation process’. Steps that were coded into this theme tended to revolve around establishing support structures and processes for tool use, looking to teacher, peer and external sources for input.


I would ask the teacher to propose some approaches to support her development. (Workshop 3)

Availability: Consider other systems / supports: who would be available to support me? (Workshop 2)

‘Implement’ was another common theme that emerged from the analysis. Groups tended to describe steps that were coded into this theme towards the end of the tool selection process. A broad range of actions were coded into this theme, although these steps most commonly centred on providing support to teachers during implementation, often defined in terms of ongoing mentorship or the provision of feedback.

[Teacher] has resources and strategies to share with mentor teachers he is working with. (Workshop 2)

I would meet with [the teacher] regularly along with other trial participants to support them to document how they have gained feedback and used it to identify professional learning. (Workshop 3)

Regarding how groups ordered their steps for tool selection, a few key points emerged from the analysis. Firstly, there were general similarities between groups as all four workshops followed roughly the same order, starting the process by considering the context, determining tool applicability, setting up the implementation process and finishing with implementation. However, as the table overleaf shows, tool selection processes across all groups did not typically follow a linear process. For example, Workshop 2 participants began their process of tool selection by investigating available tools, determining tool applicability, then use, moved on to consider the context, but then returned to ‘investigate tools’ two steps later. This cyclical process was typical of all four workshop groups, as steps in tool selection would move towards implementation.

Having categorised the themes into four broad categories, (initiate, set up, implement and evaluate, support systems learning), groups in all four workshops tended to identify steps for tool selection that fell into the first three stages of the evaluation model, initiative, set up and evaluating. Only Workshop 2 participants identified a step that involved supporting system learning. However, it should be noted that workshop participants were not explicitly asked to extend their steps beyond tool selection.

Table 14. Tool Selection Process Themes

Step

Workshop 1 Workshop 2 Workshop 3 Workshop 4

1 Consider context Investigate tools Consider context Discuss Consider context

2 Discuss Consider context

Determine tool applicability


Discuss Set up implementation process

3 Determine tool applicability

Discuss Determine tool use



4 Determine tool applicability Tool validity and alignment with APST

Consider context CollaborateSet up implementation process

Investigate tools

5 Collaborate Implement

Determine tool applicabilityConsider context

Set up implementation processMonitor progress


6 Identify goalsPlan data-use

Investigate tools Consider implementation resources

Implement

7 Consider implementation resources

Set up implementation process

Monitor progress Investigate tools


8 X Consider implementation administration

ImplementMonitor progress


9 X Implement X Set up implementation process

10 X Share process X X*Workshop 1 was held in Sydney with 10 participants. Workshop 2 was held in Darwin with 5 participants. Workshop 3 was held in Melbourne with 19 participants. Workshop 4 was held in Perth with 13 participants.

Table 15. Legend for Tool Selection Process Themes

Evaluative process category

Focussing

Initiating

Evaluating

Supporting learning

Participants were also asked to reflect on broader questions to elicit additional considerations associated with tool selection. A summary of responses to each question is provided in the table below.

Table 16. Workshop Participants' Considerations for Selecting Tools

Question Comment summary

Does it suit your purposes for assessment?

The selected tools must be ‘user-friendly’, relevant to supporting further understanding of a teacher’s strengths and weaknesses and had practice domains that were aligned or related to the APST.

Would the information generated by the tool be sufficient for the identified purpose?

The tool needs to generate structured clear feedback. It does depend on whether the assessment is applicable to an individual or a whole school context. For example, TSAT might be sufficient for the assessment of individual learning but may not hold much relevance towards overall school improvement plans.

What other information do you imagine you would need in order to use the appropriate tool(s)?

Other resources about using the tools might be helpful. For instance, guidance on giving feedback from the tool to the teacher, and for the teacher to receive feedback.

What would you do with the information generated from using the selected tool(s)?

Information could be used to support mentoring and peer-to-peer collaboration, or as a baseline for monitoring and reflection on progress or tracking growth competencies. Use of any data from this tool also needs to be considered, for example how it would be used by school leadership.

Across the workshop activities, the comments provided by participants highlighted an importance for tools, resources and evaluation practices for assessing professional growth to be highly relevant to context, teacher career stage, and of secondary importance, for implementation considerations to factor into the decision about use, including available resources (eg funds to purchase tool), student population characteristics, teacher capability and area of need, and familiarisation with the tool and ease of use.


There was also emphasis placed on the teacher’s own knowledge of their areas of strength or needs, where the impetus to drive learning and growth is placed on them as a teacher. Concerns raised focused on factors that would affect implementation or appropriate use of any identified tool. These included protected time as a resource, particularly regarding the potential for teachers to engage in collaborative discussions with school leaders about growth opportunities, having protected time to administer and use the tool, and the potential uses and ownership of the data generated by assessment tools and evaluation practices.

3.5. Delphi SurveyThe modified Delphi survey was administered simultaneously with the stakeholder workshops. The first round was from May 21st – June 27th 2019, and the second and final round from July 15th – August 28th 2019. The following section presents the results of responses to both rounds of the Delphi survey. More detailed overviews of survey data are included in Appendix D.

The main focus of the Delphi surveys was to test the principles generated from the literature scan and individual and small group stakeholder consultations.

3.5.1. Participant Characteristics A total of 70 participants completed the survey across both rounds. The table overleaf shows the role breakdown of participants, where a large proportion were those who identified as school-based staff. In the first round, the highest number of participants were school teachers (22.7 per cent), closely followed by principals (20.5 per cent), in contrast to the second where the majority of participants were teacher regulatory authority staff and education consultants.


Table 17. Roles of Delphi Survey Participants (Round 1 and 2)

Role Round 1

n

Round 1

Percentage

Round 2

n

Round 2

Percentage

School teacher 10 22.7% 1 3.85%

Principal 9 20.5% 2 7.69%

Other5 8 18.2% 7 26.92%

School leader 5 11.4% 1 3.85%

Initial teacher educator 4 9.1% 3 11.54%

Policy maker 3 6.8% 2 7.69%

Assistant principal 2 4.5% 2 7.69%

Teacher regulatory authority staff 2 4.5% 7 26.92%

Researcher 1 2.3% 0 0%

NA 0 0% 1 3.85%

TOTAL 44 100% 26 100%

The table below and overleaf shows the numbers and percentages of participants by organisation. It should be noted many participants indicated they worked in multiple organisations, so the total number in the organisations represented in the table below is greater than the number of participants completing the survey (n=70). Many participants among those who indicated multiple organisations belonged to principal associations and government schools, with a combination of other organisations.

Table 18. Organisations of Delphi Survey Participants (Round 1 and 2)

Organisation Round 1

n

Round 1

Percentage

Round 2

n

Round 2

Percentage

Principal association 8 6.3% 1 3.33%

Independent school 8 6.3% 0 0%

Government school 7 5.5% 3 10.00%

Union 6 4.7% 0 0%

Teacher association 5 3.9% 3 10.00%

Professional association 5 3.9% 3 10.00%

Government Department of Education 5 3.9% 6 20.00%

Initial teacher education institution 4 3.1% 1 3.33%

Catholic school 4 3.1% 1 3.33%

Regulatory authority 3 2.3% 7 23.33%

Non-government research or policy organisation 3 2.3% 0 0%

5 The three participants who selected “Other” indicated that they worked in an education consultancy role.


Organisation Round 1

n

Round 1

Percentage

Round 2

n

Round 2

Percentage

Other6 3 2.3% 4 13.33%

Independent school office 2 1.6% 0 0%

Catholic Education Office 1 0.8% 0 0%

N/A 0 0% 1 3.33%

TOTAL 64 100% 30 100%

The table below shows the primary location of participants’ current work. The largest groups were from New South Wales (10.3 per cent) and South Australia (9.6 per cent) in the first round; however, in the second the largest number of participants were from Victoria (31 per cent) and the Northern Territory (15.4 per cent). Several participants indicated they worked in multiple jurisdictions, and therefore the total number in the locations is greater than the number of participants (n=70).

Table 19. Primary Geographic Location of Delphi Survey Participants (Round 1 and 2)

Location Round 1n

Round 1Percentage

Round 2n

Round 2Percentage

New South Wales 15 20.5% 2 7.7%

South Australia 14 19.2% 3 11.5%

Western Australia 10 13.7% 2 7.7%

Victoria 9 12.3% 8 30.8%

Queensland 9 12.3% 1 3.9%

Tasmania 6 8.2% 1 3.9%

Australian Capital Territory 4 5.5% 3 11.5%

Northern Territory 3 4.1% 4 15.4%

International 3 4.1% 1 3.9%

National 0 0% 1 3.9%

N/A 0 0% 1 3.9%

TOTAL 73 100% 26 100%

3.5.2. Principles of Conducting Evaluation in a System that Supports Professional Growth

As noted earlier, respondents were asked to give their perspectives on the eleven principles identified through the literature scan. Specifically, they were asked to rank the importance of each principle relative to one another (where principles were broken down into a number of smaller, specific items). Mean scores of importance rankings were calculated to identify the average order of importance of the principles.

As this was a Delphi survey, the results of the first round of data informed the design of the second round of the survey. Altogether, a total of 20 items were removed following analysis of the first round of the survey, as these did not meet the consensus cut-off score for the first round of the survey. These items were found within the six of the eleven principles: student-based (n=6), cost-effective (n=4),

6 Two participants who worked in ‘other’ organisations worked at a university.


comprehensive (n=4), regular (n=3), and collaborative (n=3). Respondents were asked to either endorse the removal of these items or indicate whether these should be retained within a framework for selecting teacher evaluation tools. This is presented in the table below.

Table 20. Percentages of Responses for Each Item Proposed for Removal

Principle Item Agree-Remove (%)

Disagree-Keep (%)

Cost-effective Consider infrastructure requirements 80.8 19.2

Comprehensive Use indirect evidence (e.g. reviewed later) 65.4 34.6

Comprehensive Use supplementary evidence (e.g. third-party reporting) 65.4 34.6

Cost-effective Consider analytical expertise 61.5 38.5

Cost-effective Consider costs of a specific assessment strategy or tool 50.0 50.0

Student-based Involve students in the measurement of teacher growth 50.0 50.0

Regular Be conducted to provide ongoing progress monitoring 46.2 53.8

Cost-effective Consider administrative support 46.2 53.8

Student-based Gauge and measure student enjoyment 46.2 53.8

Regular Require assessment of performance over time to assess competence

42.3 57.7

Regular Require regular accumulation of data or evidence to assess competence

38.5 61.5

Collaborative Involve high levels of communication between the assessor and individual being assessed

30.8 69.2

Student-based Involve student feedback on teacher practice 30.8 69.2

Collaborative Allow for collaboration between peers 26.9 73.1

Student-based Involve students in the feedback/evaluation process 26.9 73.1

Comprehensive Use direct evidence (e.g. witnessed by assessor) 26.9 73.1

Comprehensive Use self-assessment 26.9 73.1

Student-based Gauge and measure student engagement 23.1 76.9

Collaborative Allow for collaboration between the assessor (e.g. Principal) and the practitioner

15.4 84.6

Student-based Gauge and measure student interaction 15.4 84.6

A small number of items received a high percentage of responses supporting removal from the framework, with ‘high’ taken to mean more than 60 per cent of responses). These were related to the principles of cost-effectiveness (that selection of a resource should consider infrastructure requirements), and comprehensiveness (that a resource or method provides indirect evidence, and supplementary evidence). However, responses were divided for all other items, signifying that participants still considered these important for resource selection.

3.5.3. Mean Rankings: Round 1 and Round 2The weighted mean rankings of principles across Round 1 and Round 2 of the survey are presented in this section. Round 1 and Round 2 means are the average of the means of all items comprised within a principle. The Round 1 and Round 2 SD values are the standard deviations of the means of items. The overall mean and overall SD are the mean, and standard deviation of the weighted means of items across


two rounds. Means were calculated directly from the two rounds, with items removed from Round 1 excluded in the weighting for the purposes of ensuring a representative comparison between the two rounds.

Results are presented in the table below, with the principles ranked by highest overall mean score to lowest overall mean score. Across both rounds, the principle of ‘understandability’ of the purpose, standards, data and feedback from the assessment was deemed to be of most important; furthermore, participants indicated that it was important for the tool to allow for professional growth to occur (for example, through the data that is provided). Of lowest importance was that tool has utility for the teacher, and that it allows for teachers to have professional agency, or in other words, take ownership of their professional growth journey. However, it should be noted that participants did not necessarily view these principles as not important, but that these were of lesser importance relative to other principles.

Table 21. Mean Scores for Principles Weighted Across Survey Rounds

No. Principle Items (n) Round 1 Round 2 Overall

Mean (SD) Mean (SD) Mean (SD)

1 Understandable 5 9.49 (0.13) 8.79 (0.29) 9.22 (0.18)

2 Professional growth 4 8.95 (0.21) 8.34 (0.29) 8.71 (0.24)

3 Evidence-based 7 8.80 (0.19) 8.51 (0.27) 8.69 (0.18)

4 Authentic 7 8.76 (0.43) 8.08 (0.69) 8.56 (0.28)

5 Relevant 2 8.74 (0.33) 8.02 (0.68) 8.51 (0.52)

6 Context-specific 3 8.46 (0.28) 8.31 (0.10) 8.46 (0.46)

7 Collaborative 6 8.32 (0.56) 7.91 (0.75) 8.40 (0.18)

8 Comprehensive 7 8.19 (0.60) 8.32 (0.04) 8.27 (0.30)

9 Regular 4 7.83 (0.35) 7.35 (N/A) 7.86 (0.31)

10 Cost-effective 5 7.68 (0.56) N/A 7.68 (0.56)

11 Student-based 9 7.58 (0.75) 7.71 (0.39) 7.40 (N/A)

12 Professional agency 10 N/A 7.30 (0.70) 7.30 (0.70)

13 Utility 7 N/A 6.35 (1.02) 6.35 (1.02)

In addition to mean scores, the degree of convergence between rankings was also examined using inter-rater reliability analysis, the intraclass correlation coefficients7 (ICC) and Krippendorff’s Alpha Reliability, was calculated using R (Gamer et al., 2013). This indicates the level of agreement between respondents regarding their perceived importance of principles8.

The intra-class correlation coefficients (signifying level of agreement) and degree of reliability were calculated for each of the items within each principle. As displayed in the following tables, the overall reliability of the data is rated ‘good’. Survey respondents have high agreement on the importance of the items in the principles of comprehensiveness, student-based, collaboration, authenticity, and cost

7 The ICC used here is based on a mean-rating, absolute-agreement and 2-way random-effects model.8 Values less than 0.5 are indicative of poor reliability, values between 0.5 and 0.75 indicate moderate reliability, values between 0.75 and 0.9 indicate good reliability, and values greater than 0.90 indicate excellent reliability (Koo & Li, 2016).


effectiveness. There is moderate agreement on the importance of the items in the relevant and context-specific principles.

In Round 1, there was low agreement on the importance of the items in the principle of professional growth, regularity, understandability and evidence-based teacher professional growth assessments. Low reliability of this data could be attributed to the perceived high importance of all items within the principles, meaning that it is difficult for respondents to determine the exact importance of each item.

A good reliability was found for the rankings between respondents in Round 1. The average ICC was 0.866 with a 95 per cent confidence interval from 0.718 to 0.957, which is significantly different from a random ranking (F (10, 420.6) = 6.901, p < .001).

Table 22. Intra-class Correlations for Each Principle and Overall (Round 1)

Dimension ICC ICC lower bound ICC upper bound p Degree Items (n) Respondents (n)

Comprehensive 0.86 0.68 0.97 < .001 good 7 42

Student based 0.84 0.67 0.95 < .001 good 9 41

Collaborative 0.82 0.57 0.97 < .001 good 6 43

Authentic 0.78 0.51 0.95 < .001 good 7 43

Cost effective 0.77 0.44 0.97 < .001 good 5 42

Relevant 0.63 -1.14 1.00 = 0.112moderate

2 41

Context specific 0.57 0.01 0.99 = 0.024moderate

3 43

Regular 0.55 0.11 0.96 = 0.007moderate

4 41

Professional growth

0.49 0.06 0.95 = 0.011 poor 4 41

Understandable 0.44 0.06 0.90 = 0.01 poor 5 41

Evidence based 0.25 -0.09 0.75 = 0.08 poor 7 42

Table 23. Intra-class Correlations for Each Principle (Round 2)

Dimension ICCICC lower bound

ICC upper bound

p Degree Items (n)Respondents (n)

Collaborative 0.84 0.43 1.00 < .001 good 3 26

Utility 0.80 0.56 0.96 < .001 good 7 26

Authentic 0.79 0.53 0.95 < .001 good 7 26

Removal 0.74 0.56 0.88 < .001 moderate 20 26

Relevant 0.71 -0.34 1.00 = 0.058 moderate 2 26


Professional agency 0.69 0.42 0.90 < .001 moderate 10 26

Understandable 0.43 0.04 0.90 = 0.014 poor 5 26

Student based 0.41 -0.03 0.98 = 0.038 poor 3 26

Professional growth 0.38 -0.08 0.93 = 0.057 poor 4 26

Evidence based 0.22 -0.18 0.76 = 0.143 poor 7 26

Context specific -0.17 -0.30 0.80 = 0.661 poor 3 26

Comprehensive -0.20 -0.22 0.35 = 0.908 poor 3 26

Regular 1 26

A good reliability was found for the rating between the respondents. The overall ICC for all 56 items was 0.846 with a 95 per cent confidence interval from 0.776 to 0.9, which is significant different from a random ranking (F(55, 198.7) = 9.223, p < .001).

Given the number of participants in Round 2 is small (n=26), the overall ICC in Round 2 is lower than the previous round. However, the 95 per cent confidence intervals in Round 1 and Round 2 overlapped, meaning they are not significantly different.

It should be noted that while definitions for the principles were presented to survey respondents in the form of hover-text and a downloadable Microsoft Word file, it is possible that respondents responded to these items based on their perceived understanding of each of the items, rather than the definitions provided. For this reason, item-level analysis was done, that is respondents’ perspectives on the importance of components within each principle was also analysed which is provided in Appendix D.

3) OPEN-ENDED COMMENTS Respondents were also asked across both surveys to provide a response to eight open-ended questions. The table below details common responses to these questions, and whether such responses were consistent across both rounds of the survey.

Table 24. Summary Responses to Open-Ended Delphi Survey Questions

Open ended question Responses Round 1 Round 2

Looking at the above and accompanying descriptions [the principles], what, in your opinion, is redundant for guiding assessments to support teacher professional development?

Principles are comprehensive ✔

Cost-effectiveness ✔

There is overlap between principles ✔

There is a concern that student feedback carries biases ✔

Looking at the above and accompanying descriptions [the principals], what, in your opinion, is missing for

Assessments don’t target teachers’ beliefs, something that would otherwise support them to better determine areas for professional development.

✔ ✔



guiding assessments to support teacher professional development?

Assessments are not tailored to teachers needs and goals ✔

Assessments are not designed for professional development to be teacher-driven

✔

Time is needed as a pre-condition for effective teacher evaluation

✔

Trust and a supportive school culture are pre-conditions for effective teacher evaluation

✔

Based on your past or current experience, how do you identify teachers’ professional growth needs?Professional growth relates to your own, or that of teachers.

By linking them to standards ✔ ✔

Through collaboration, teacher observations, feedback, and mentor relationships

✔ ✔

Through balancing the needs of both the school and the teacher

✔

Not by PD conducted for compliance purposes ✔

Based on your previous or current experience, what measures or systems do you/your colleagues use to identify whether professional growth is occurring?

Teacher observations and feedback ✔ ✔

Teacher confidence ✔

Systems that align to the APST Standard ✔

Collaborative systems that draw on a range of data sources.

✔

How can you/your colleagues be best supported to generate useful and actionable information toguide professional growth of teachers?

Provide teachers with time to conduct PD ✔ ✔

Provide teachers with more training in data-use ✔

Establish collaborative systems that are tailored to the individual and school context

✔

Ensure evaluation is supportive, not designed to sanction teachers

✔

In your opinion, what systems or structures need to be in place in order to supporta culture of professional growth and evaluation?

Systems that are teacher-led ✔ ✔

Structures that use standard-linked tools ✔

Systems that are embedded at the school level ✔

Systems of trust ✔

Structures that are based on a growth mindset ✔ ✔

In your view, what would Clear principles and guidelines NA ✔



need to be included in the framework to enhance current teacher evaluation in Australia? *

Information on tool-use best practice NA ✔

Evaluations to be context-specific NA ✔

Teacher and assessor training to improve teacher assessment

NA ✔

Do you have any furthercomments that you would like to share? *

Teachers’ professional growth should be holistic. NA ✔

A wide range of stakeholders, including academics, leaders, educators, and community members, should be incorporated into the process of developing teacher professional development frameworks

NA ✔

The analysis of the open-ended questions from Rounds 1 and 2 reveals a number of key findings. By comparing responses from Rounds 1 and 2, we can see a number of key themes that are shared between Round 1 and 2 participants. Both cohorts emphasised the importance of a positive and supportive school culture to successful evaluation and teacher professional development.

The significance of trust and school culture points to a second emphasis revealed by the analysis, as both round 1 and 2 participants identified collaborative systems as being key for measuring, supporting and identifying teachers’ professional growth needs, in particular, teacher observations and feedback. However, to ensure that these systems are effective in supporting teachers’ professional growth needs, it was reported by participants from both cohorts that teachers need protected time away from the classroom.

“Provide us with regular time off class to collate, document, reflect and develop useful and actionable items to guide professional growth. Ideally, a full-time teaching assistant in every class would provide teachers with the support they need in the classroom which would free up time for them to do the above.” (Education Specialist/Educator)

3.5.4. Modelling the Relationship Between PrinciplesTo support and maximise utility of the study findings, we sought to model the relationships between principles. Given the nature of the data, non-metric multidimensional scaling (MDS) was used. MDS is a visual representation of distances or dissimilarities between sets of objects. In the current case these objects are judgements of the various items. Objects that are more similar (or have shorter distances) are closer together on the graph than objects that are less similar (or have longer distances). As well as interpreting dissimilarities as distances on a graph, MDS can also serve as a dimension reduction technique for high-dimensional data (Buja et al., 2008).

MDS is used often when the sample size is too limited for other methods (such as factor analysis). MDS is a non-parametric method and the result is less sensitive to the sample size and can handle non-metric data. In the present case, nonmetric MDS is used, as this assumes the data are at least ordinal but not necessarily interval or ratio. The method was developed by Sammon (1969) because it yields a smaller stress and more defensible solutions than using Kruskal’s (1964) non-metric multidimensional scaling in our dataset. In Sammon's mapping the errors in distance preservation are normalized with the distance in the original space and because of this the normalization the preservation of small distances will be emphasised. This was accomplished using the R package “mass” (Ripley et al., 2013).


There are measures of goodness-of-fit, such as the S-Stress test. S-Stress is standardised to take values between 0 and 1. Hair, Anderson, Tatham, and Black (1998) have recommended that Stress values below 0.15 represent a good fit and also suggest Dispersion Accounted For (DAF) and Tucker's Coefficient of Congruence should have values 'close to 1' (cf Dugard, Todman, & Staines, 2010, p. 275).

To evaluate the optimal model, a 3- 4-ad 5 cluster model was specified using the items from Round 1. The figures below present the items in 2-dimensional space, and the S-Stress of .09 indicated excellent fit of the data to this model. These were then clustered, and the 3-cluster solution provided the most defensible fit (see Figure 12).

Figure 11. Scattergram of the principles at the item level


Figure 12. 3-Cluster solution for the principle at the item level.

Figure 13. Scattergram for the principles.


Figure 14. 3-Cluster solution for the principles with dimension groupings.

The groupings of the principles outlined in the 3-cluster solution indicate that there are three higher-order dimensions being captured, which can be interpreted as content, purpose and process.

Table 25. Dimension Groupings of Principles

Dimension Description

Content The content of the evaluation, assessment or feedback resource, or in other words what is being captured in the evaluation, assessment or feedback tool or resource. The four principles in this dimension (relevant, understandable, authentic, and context-specific) describe the characteristics of content that should be captured in an evaluation, assessment or feedback resource that could be used effectively in a system that supports professional growth.

Purpose The purpose or intent of the evaluation, assessment or feedback resource (collaborative, evidence-based, professional growth). The three principles in this dimension describe the intentional characteristics of evaluation, assessment or feedback that would support teacher professional practice growth.

Process The process of the evaluation, assessment or feedback event. The three principles in this dimension (regular, student-based, comprehensive) describe the characteristics of conducting an evaluation, assessment or feedback in a way that should support teacher professional practice growth.

Product The purpose of evaluation or tool relates to the development in teacher practice and ultimately student outcomes. The combination of these dimensions is aimed at enhancing teacher practice and student outcomes, both of which can be considered the product of the evaluation, assessment or feedback event.


The findings of the MDS are logical, and interestingly are similar to a widely-known educational evaluation framework the Context, Input, Process and Product model (Stufflebeam, 1983). Operationally, grouping the principles by these dimensions may also be more practical given that it enables educators and other stakeholder to focus on three key priority areas when selecting and planning for an evaluation that promotes teachers’ professional growth.

Overall the findings of the Delphi survey indicate that there is relatively high level of consensus of the appropriateness of the principles for guiding evaluation, assessment or feedback to promote teachers’ professional growth. Further, that the enactment of said principles requires the establishment of a system underpinned by pre-conditions. The following section discusses the key findings in more depth, including a triangulation of the convergence of findings across data sources and the strength of evidence for each finding.


4.Triangulation and SynthesisThe results of the literature scan, individual and group consultations, stakeholder workshops, and Delphi survey produced three key components of the Framework:

1. Preconditions for an evaluation system that promotes professional growth2. Endorsed and widely tested principles to guide assessment, evaluation and feedback practices

within a system that promotes professional growth3. A process model including detailed operational steps developed by stakeholders to conduct an

evaluation based on a teacher’s professional growth need(s)

In each of the three components, the key findings have been triangulated and the strength of evidence determined across the data sources.

Preconditions for Evaluation SystemsThe requirement for preconditions (categories) for an evaluation system that supports teachers’ professional growth was supported unequivocally by all stakeholders and across all data sources. With regards to what these preconditions are or should be, there was convergence in findings across data sources for most of the categories of conditions, with the exception of ‘evaluative mindset’. This is not to say that the term ‘evaluative mindset’ was not used by interviewees and stakeholder workshop attendees, but rather the concept(s) and processes underpinning such a mindset were not as evident in these discussions.

Table 26. Triangulation Matrix for Preconditions

Pre-conditions Sources Strength of evidence

Quantitative

(Delphi Survey)

Qualitative

(Interviews, Stakeholder workshops)

Evidence

(Literature)

(***high, **med, *low)

Shared purpose, explicit criteria ***

Relational trust ***

School leadership, positive school climate ***

Evaluative mindset **

For most of the preconditions the strength of evidence is high both within and across sources, meaning that, for example, there was no reviewed literature that stated that shared purpose or explicit criteria was unimportant, or inappropriate for supporting teachers to build their professional growth. The strength of evidence for evaluative thinking is moderate, largely because it is a relatively more recent concept (at least in the literature), and thus there is not yet a great deal of empirical evidence that demonstrates the importance of such a mindset for the Australian teaching population. However, early evidence is highly compelling and posits that an evaluative mindset can sit within a teacher’s decision-making and judgement process.


Principles for Selecting Evaluation ResourcesThe table overleaf maps the key findings for each principle along with a strength of evidence rating (includes convergence of findings and quality of data gathered).


Table 27. Triangulation Matrix for Principles for Guiding Assessment, Feedback and Evaluation Practices

Principle Literature Workshops Delphi survey Strength of evidence

Understandable

Assessment, evaluation or feedback must be understandable to the teacher being evaluated.

Verified Understandability is critical. However, not all teachers will have the same level of understanding about evaluation, assessment or feedback.

Respondents indicated the highest level of importance of this principle in comparison to all others (overall M=9.22[0.18]). With a low level of agreement among stakeholders in importance ratings. MDS results indicated this principle is related to context-specific, relevant and authentic.

**

Professional growth

Assessment, evaluation or feedback must be conducted with an intention to support professional growth AND must generate information that can support professional growth.

Verified Further assessment, evaluation or feedback must be specific to a teacher’s individual professional growth needs and personal goals.

Respondents indicated the second highest level of agreement with this principle (overall M=8.71(0.24). Further, it was rated of high importance in comparison to all other principles. MDS results indicated this principle is related to collaboration and evidence-based and falls within the purpose dimension.

***

Evidence based (Assessor expertise)

Assessment, evaluation or feedback must be carried out by professionals with the relevant expertise in the skills being assessed with a high level of knowledge of the expected standards. Further judgements made in the assessment, evaluation or feedback must be justified based on relevant evidence.

Verified It is important that the assessor has personal professional experience that is similar or highly related to the individual who is being assessed or evaluated. In practice, this means teachers and school leaders are well placed to conduct evaluation of teachers.

Respondents indicated a high level of importance of this principle (overall M=8.69[0.18]). With a low level of agreement among stakeholders. MDS results indicated this principle is related to collaboration and professional growth which fall within the dimension of ‘purpose’.

**



Authentic

Assessment, evaluation or feedback should occur within the context or setting of practice of the person being evaluated to capture accurate evidence of skills, attitudes and knowledge. Evidence gathered needs to reflect real-life practice and needs to be collected in a manner that ensures honesty and validity.

VerifiedStakeholders noted this is also more likely an achievable when the assessor or evaluator is also based in the context or setting of practice of the person being evaluated.

Respondents indicated a moderate level of importance of this principle in comparison to others (overall M=8.56 (0.28)).Similarly, there was a moderate level of agreement among stakeholders. MDS results indicated this principle is related to relevance, context-specific and understandable. Further it falls within the dimensions of ‘content’.

**

Relevant

Assessment, evaluation or feedback should be based on a diverse set of profession-specific standards, competencies or performance metrics which should be identified in relation to the context of practice.

VerifiedStakeholders noted that the APST and the ATPDF are relevant here, but also prefaced this with a need to ensure that the context of practice is taken into account when applying these standards.

Respondents indicated a moderate level of agreement and moderate importance for this principle (overall M=8.46 (0.46)).MDS results indicated this principle is related to related to context-specific and understandable. Further it falls within the dimensions of ‘content’.

**

Context-specific

All assessment, evaluation or feedback (regardless of tool or resource) should be applicable and where appropriate tailored to practice settings, individual teacher’s priorities, and school priorities and contextual characteristics.

VerifiedStakeholders highlighted again how related this principle is to almost all of the others. They also highlighted that there is a tension between the importance of ensuring assessment, evaluation or feedback is specific to school context and priorities along with the teachers’ own professional growth priorities.

Respondents indicated a moderate level of agreement and moderate importance for this principle (overall M= 8.40(0.18)).MDS results indicated this principle is related to related to related to relevant, understandable and authentic. Further it falls within the dimensions of ‘content’.

***



Collaborative

Assessment, evaluation or feedback should involve high levels of communication, reflection and reinterpretation of performance. May include collaboration between the assessor or school leader and the practitioner to enable validation of judgements made or may involve collaboration between peers.

Verified However, stakeholders identified that the preconditions in particular need to be in place to ensure this supports professional growth (supportive leadership and culture that supports teacher learning) is required for the collaboration to be beneficial.

Respondents indicated a high level of agreement, with a moderate level of importance (overall M=8.27 (0.30)).MDS results indicated this principle is related to evidence-based and professional growth, falling within the dimension of ‘purpose’.

***

Comprehensive

Assessment, evaluation or feedback should involve multiple samples, or multiple assessors and multiple methods. Further evidence can be direct, indirect or supplementary (from a third party).

VerifiedStakeholders also highlighted the relationship between comprehensiveness and authenticity.

Respondents indicated a high level of importance of this principle (overall M=8.27 (0.30)). Agreement among participants on this principle was the highest of all principles. MDS results indicated this principle is related to student-based and regular; it also falls within the dimension of ‘process’.

***

Regular

Assessment, evaluation or feedback should be regular, which may involve accumulating data or evidence about performance over a period of time. Regular assessments may provide ongoing progress monitoring and yield feedback on a variety of specific practices.

Not verifiedStakeholders held mixed views about this principle, highlighting a distinction between timeliness, and repeated assessment, evaluation or feedback. A number of implementation considerations including workload and resourcing were discussed as potentially off-setting the benefits of repeating assessments. However, timeliness was considered important, where assessment, evaluation or feedback should occur at a time when the teacher is going through a process of re-setting goals, or self-reflection where they can quickly act on the information to improve their practice.

Respondents indicated a low level of agreement and a low level of importance of this principle in relation to all others (Overall M=7.86(0.31)). MDS results indicated this principle is related to student-based and comprehensive, falling within the dimension of ‘process’.

**

Student-based Assessment, evaluation or feedback should be Not verified Respondents in Round 1 indicated a **



designed to gauge and measure student growth, engagement, interaction, enjoyment or understanding. These might involve students in the feedback/evaluation process, measurement of growth or entail observation of classroom interactions.

Stakeholders offered mixed views on this principle. They felt in most cases directly involving students in the feedback, assessment or evaluation of teachers would be challenging. However, they thought that student feedback is highly beneficial to include in any assessment, evaluation or delivery of feedback to teachers.

high level of agreement, with a moderate to low importance in relation to other principles (Overall M=7.40(0.70)).The principle was suggested for removal in Round 2, and received 46 per cent endorsement for removal. MDS results indicated this principle is related to regular and comprehensive, falling within the dimension of ‘process’.

Professional agency

Not identified in the literature scan This principle was identified through the early stakeholder workshops in response to the results of Round 1 of the Delphi survey. Stakeholders explained that this principle was important because a distinction and balance is required between assessment, evaluation or feedback being aligned and specific to a teacher’s personal professional growth needs and the school improvement priorities or state and territory priorities for improving education. Practically, the principle is also important for ensuring that a teacher has the agency to work on and pursue their own professional growth goals irrespective of what school context they are working on.

The principle was included in Round 2 of the Delphi survey. Respondents rated that in comparison to others, this principle was less important: M=7.30 (0.70). Further, there was a low level of agreement among stakeholders about the importance of this principle.

*

Utility Not identified in the literature scan This principle was identified through the early stakeholder workshops in response to the results of Round 1 of the Delphi survey. Stakeholders explained that utility is related but still distinct from the other principles. It indicates that the process and products from an assessment, evaluation

The principle was included in Round 2 of the Delphi survey. Respondents rated that in comparison to all other principles, this was the least important M=6.35 (1.02).

*



or feedback resource need to be useful, which means they need to be relevant to context, specific to the individual’s professional growth goal, and actionable for use in informing practice in relation to the teacher’s immediate goal or problem of practice.


Based on the results of the triangulation and the key findings, we operationalised the two outputs of the study; (i) preconditions for building a system that promotes teachers’ professional growth, and (ii) the principles of selecting assessment, feedback or evaluation resources into an implementation plan, referred to as the Evaluation Process Model (overleaf). As indicated in the figure overleaf, it is our suggestion that there are four phases in the Evaluation Process Model:

1. Focussing2. Initiating3. Evaluating4. Supporting learning

Part B of this report entails an implementation plan for the Evaluation Process Model for schools, and the role of teachers and leaders in this process. Each stage is described and applied to a real-life scenario, along with links to resources to support the application of this model in an Australian education setting.

To end the report (Part A), the following section discusses the key conclusions and recommendations for supporting teacher evaluation that promote professional growth in the Australian education system based upon the findings of this consultative study.


Figure 15. Evaluation Process Model.


5.ConclusionsThere is little doubt that effective evaluation systems of teachers and teaching can enhance continuous and targeted professional learning while at the same time supporting teachers to become evaluative thinkers and develop or adapt their practice through feedback. Such evidence has grown to the extent that evaluative practices that support a focus on evidence informed practice by educators are now considered essential for school improvement. It is this thinking that informed the foundation of this study.

Subsequently, this report provides an overview of the research activities undertaken and the overall results of the Teacher Evaluation Framework study. In addition, insights into the development of an evaluation implementation model are suggested. It is essential to note that the resulting Evaluation Process Model is designed and positioned to support and operationalise the implementation of the Australian Teacher Performance and Development Framework. Ultimately it was constructed to support school teachers and leaders to implement an evaluation system that is fit-for-purpose and therefore support improvement in teaching practice.

The identification of necessary pre-conditions for the establishment of a teacher evaluation associated with professional growth was a key aspect of this study. The identification and testing of a principled based approach to assessment and evaluation practices for teachers’ professional growth in Australia, as well as the steps and processes to support evaluation planning and resource selection were a key focus of this study.

The generation of the Evaluation Process Model highlighted key factors for a success criterion for the implementation of the Evaluation Process Model within the Australian context. Importantly, it is evident that a collective and coherent approach is essential. The diagram below depicts a possible success criterion across the Australian education system, for school leaders and school contexts, and teachers. This diagram was built from the voice of the evidence and current evidence but is by no means exhaustive. The success criteria suggest the responsibilities for the key stakeholders should not be viewed in isolation but as a collective within a very complex and connected system where each group takes a responsible role.

The educational system here is defined as central departments either at Commonwealth, State or Territory or Regional level provides the foundational policy and direction infrastructures, while the context of the school and leadership provides the infrastructure to implement and the teacher of course, are at the centre of these activities. What is seemingly essential across all is efficiency in infrastructures, implementation and a mindset change. We would also argue that the system needs to promote a consistent, useful, feasible, accountable and contextually appropriate approach to the evaluation of teachers. The diagram below highlights those key responsibilities for such a system.


Figure 16. Success criteria for the implementation of the Evaluation Process Model.

We would argue that further support to underpin and strengthen the current policies and evaluative systems is needed to ensure that these criteria can be achieved. For teacher evaluation practices to be utilised effectively to support professional growth across the systems we suggest a number of focussed recommendations.

1. Adopt an evaluative thinking mindset

The mindset needs to shift towards evaluative thinking becoming commonplace and being a priority, which could empower all educators within the system. Evaluative thinking is now a key competency of school leaders (Australian Institute for Teaching and School Leadership, 2014; Centre for Education Statistics and Evaluation, 2016). In the field of evaluation, especially among people involved in evaluation capacity building (ECB), evaluative thinking is increasingly recognised as a key component of evaluation capacity and high-quality evaluation practice.

Buckley, Archibald, Hargraves, and Trochim (2015) propose the following succinct definition: “Evaluative thinking is critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action” (p. 378).

Hattie and Zierer (2017) suggest that teachers need to be evaluators of their impact. Evaluation is activity that determines the merit and or worth of an event whereas evaluative thinking is a way of being a mindset. This is a state where teachers are constantly reflecting and assesses progress of not only their role in the learning lives of students but also of their own professional progress.

2. Teacher assessment and evaluation for professional growth needs to generate actionable feedback

The Evaluation Process Model and results of the study highlight the need for teacher evaluation to generate actionable, relevant and useful feedback for professional growth. In other words, information generated through assessment and evaluation needs to enable and inform evidence-based practice change in alignment with an individual’s professional growth goals. This necessitates that teachers will regularly go through a self-reflective process, utilise resources to help identify their professional growth


SYSTEMSupport flexibility for context (e.g. rural and remote)Support adaptable & high quality trainingConsistent communication & use of shared languageSupport and use dataGenerate cost-effective resourcesSupport provision of evaluationMindset shiftSupport implementation of relevant policy

LEADERSHIP & SCHOOLCapacity buildingBuy-in from leadershipInfrastructure to support implementationCost-effective resources given priorityConsistency in deliveryData useContinuity of deliveryAppropriate conditions in placeMindset shift

TEACHERSEngagement in the processBuild capacity to think differentlyUnderstand capable learnersMindset shiftBuy-in from teachersGather data & communicateCost-effective resources given priority

goals, and ensure that the goals identified are obtainable through modifiable practice change. This is where the APST, professional career stages, and the ATPDF have a significant role to play.

3. Implementation needs to be adequately resourced and valued

Broader implementation factors also fall into this area, such as the need for protected time and continuity of process. Stakeholders discussed that school leaders need to take responsibility for ensuring the preconditions are in place to enable teacher assessments and evaluations to occur, and teachers need to increasingly ensure they see themselves as active agents in the process. Teachers and educators also need to have the time to reflect on their goals and trial new or different evaluation practices to find what works for them, in their context and circumstances.

Across participating stakeholders, there was considerably rich discussion regarding how cost-effectiveness and broader implementation considerations should be factored into the process of selecting resources and evaluation planning for teachers’ professional growth, and broader teacher evaluation practices. It appeared there were some conflicting ideas, often among the same stakeholders, many of whom tended to agree that cost and broader implementation was important to think about when planning for teacher evaluation, however, cost was a comparatively less important ‘principle’ for effective teacher assessment and evaluation for professional growth. To support an evaluative system, implementation and infrastructures needs to be prioritised and given time, while being appropriately resourced. Prioritising also means ensuring that there is the expertise in place to implement and the values process.

4. Evaluation capacity strengthening, resourcing and promotion

As suggested any reform and its ensuing implementation needs resourcing. Support at all levels of the education system is pivotal to the development of an evaluative model. This study demonstrated that a mindset shift is necessary along with the development an effective and efficient infrastructure to support implementation of an evaluative model or system. This support needs to take the form of building evaluative capacity of both teachers and leaders, promotion of the value of evaluative thinking as well as promoting the merit and worth of the model or system and finally, strengthening the availability of quality evaluation resources.

Building workforce expertise in this area is essential. Better Evaluation (2019) describes this as, ‘capacity strengthening’. It is also suggested that this involves more than just one-off training, it requires a broader and multi-pronged approach.

Currently, there are very few accessible resources available in this evaluative space and communication is very much related to the notion of informing the workforce and building capacity. Evaluation as an entity comes with a stigma, social promotion of the benefits of feedback and professional growth can break this down, hence, it is suggested that a social promotion campaign could add value. Lastly, our study suggests that more resources, that are readily accessible to teachers and leaders, are necessary to support the implementation of such a system and create a mindset shift and importantly, to understand how to target professional learning to support continued professionalisation of education and enhance the expertise of teaching.

5. Evaluation practices should be teacher-driven and support of professional autonomy

One of the strongest themes that emerged from the study is the necessity for teacher evaluation practices to be teacher-driven. This goes beyond the common goal of supporting ownership and speaks more to what it means to be a teacher as a professional, which is much more explicit in the APST.

Stakeholders spoke of the false dichotomy, that often appears real, between school improvement or strategic goals and those of the teacher as an autonomous professional. Historically, teacher performance and perhaps more broadly the performance of the education system, has relied on


student outcomes as an indicator of high performance, however, this is quite removed from system inputs, and is less so but still several steps removed from the teacher and their practice. While supporting student learning outcomes is still a central purpose of the education system, evaluating teachers solely using this outcome would not only be doing a disservice to them, but also would offer little in the way of actionable feedback for professional growth. The same rationale was shared when stakeholders were explaining that teacher assessment and evaluation for professional growth must be supportive of professional autonomy rather than a school’s overall strategic plan or goal, which can change from year to year. This is necessary because teachers need to be able to take their assessment and evaluation, or in other words, their professional growth journey with them, and they need to be the stewards of the evidence and of monitoring their growth. This does not suggest that schools and leadership, and educational systems aren’t involved but more that it is a shared responsibility.

6. Information generated through evaluation practice should facilitate data sharing at every level of the system

The Evaluation Process Model should support the development of policy and implementation models in various contexts of the education sector. This would enable rich data to be gathered and provide a basis for sharing of synthesised findings and insights across jurisdictions, sectors and school contexts, with the purpose of further supporting the goal of educational reform to be authentic and evidence-informed. The diagram overleaf indicates the intended and necessary flow of information from the teachers to school leaders, schools, regions, jurisdictions and national levels.

Information flow should facilitate a feedback cycle at every level of the system, such that an evaluation activity with a single teacher is a part of informing the national educational reform agenda for how to support teachers’ professional growth in Australia, and in so doing improve the quality of education for all. This not only ensures the observable merit and worth of the system as a whole but also suggests that every teacher has a voice and plays a part within it. The message is simple, expertise matters.

Figure 17. Information flow and feedback cycle in the Education System.

7. Consider Collective impact

As discussed, there is a necessity for the shared roles and responsibilities at all levels in this evaluation system. Clarity of this shared responsibility is essential (e.g. system to support notion of integrating use of evaluation, creating usable resources for the profession). Considering the idea of Collective Impact may be valuable. The concept not only offers a mantra to work to but also clarity of roles. Kania & Kramer (2011) demonstrated that successful collective impact initiatives typically involve five conditions: a common agenda, shared measurement systems, mutually reinforcing activities, continuous communication, and backbone support organisations.


Figure 18. A model of Collective Impact, taken from Kania and Kramer, 2011.

6.Final WordThis study is underpinned by two the previous studies, first, research that explored evaluation systems internationally. This report suggested that quality teaching practices within the dimensions of pedagogical content knowledge and reflective practice are related to student achievement, it also highlighted that there is an important distinction between teaching quality (practice and instruction) and teacher quality (personal attributes, competency, skill and dispositions (Clinton et al., 2016). In addition, it was suggested that the dimensions of effective teaching can be described with four categories: a) teaching; b) the teacher as a person; c) behaviour as a teacher; and d) continuous learning as demonstrated in the figure below.


Figure 19. Teacher and teaching factors that relate to educational impact.

The second research study explored those key attributes of effective teachers. The systematic review identified the key characteristics of effective teachers and examined the magnitude of their contribution to school, student, community, and teacher outcomes (Clinton et al., 2018). It provided an understanding of those characteristics of the teacher as an individual that might inform an evaluative model. Importantly it focussed the attention of the evaluation model on impact of the teacher as opposed to dispositional characters of the teacher. The figure overleaf demonstrates the model.


Figure 20. Explanatory model of effective teacher characteristics.

It is our contention that the findings from these three studies must be combined to support the idea of the development of an evaluative system that is clearly aligned to professional growth and strengthening the capacity of our teaching workforce. At the same time this work must be aligned to understanding how to target professional learning for maximum impact. Taken together, the three reports support our understanding of the nature of evaluative systems; where we should focus the attention in term of teachers and teaching and finally, how we might implement a system with the necessary pre-conditions to ensure use supported by appropriate resources. It is assumed that this all sits within the policies and reforms of the APST, together with processes outlined in the ATPDF (Australian Institute for Teaching and School Leadership, [AITSL], 2013). These are the cornerstone of the Australia model of effective teaching.

The Teacher Practice Evaluation Framework study has demonstrated there are some key facets necessary for implementing an evaluation process to support teachers’ professional growth and aligned targeted professional learning. While the pursuit of this is not at all an insignificant undertaking, it is a progression towards viewing teachers as expert professionals. In summation, perhaps the words of the director of the General Teaching Council for Scotland are most apt:

… teacher evaluation that is tied to professional learning is a right, not a privilege. (Dr Ken Muir)


Part B: Framework for Promoting Teachers’ Professional GrowthBased on the findings of the Teacher Evaluation Practice Framework study, this document has been developed and is referred to loosely as ‘The Framework’, and presents the outputs of the study:

(1) Evaluation Process Model

(2) Rubric for Identifying and Selecting Resources

The Evaluation Process Model is for a school audience, where teachers and leaders have a role in the evaluation process. The model provides a suggested scaffold to guide thinking for educators to prioritise, plan, evaluate and synthesise information. The Rubric is situated within this scaffolding model, where it serves as a practical guide for identifying and selecting resources, alongside other considerations for an evaluative system such as infrastructure, available resources, and plans for implementation.

The suggested Evaluation Process Model and Rubric will both require further development and testing to ensure they are appropriate and useful for schools.

6.1. Development of the Evaluation Process ModelThe notion of creating a process model to guide teacher evaluation was in part conceived as a result of the familiarity educators may have with the processes associated with translating curriculum priorities into classroom practices (Brady & Kennedy, 2013).

Hence, an attempt was made to parallel the Evaluation Process Model with curriculum implementation processes. Both are dynamic in nature, encouraging movement between the different elements, with decisions being made in relation to the specific contexts of schools (Brady & Kennedy, 2013). The Evaluation Process Model, as shown in the figure overleaf, is cyclical in nature, in recognition that teachers’ professional growth is a continuing activity, which not unlike the curriculum implementation process, “…[are] constantly in a state of change as new information or practices become available” (Print, 1993, p. 69).

Similar to the practice of analysing the situation in which a curriculum is being implemented is the suggested starting point in cyclical models of curriculum development (Brady & Kennedy, 2013; Print, 1993). The starting point in the Evaluation Process Model is to clarify the purpose of developing teachers’ professional growth and determine the current situation. It is important to note, however, that schools do not exist within a vacuum. Although schools have their own characteristics, they occur within overlapping national and state and territory jurisdictional policies and priorities, which is relevant to teacher evaluation and curriculum development and planning.

The development and representation of the Evaluation Process Model was also informed by a process improvement sequence often evident in many organisational development frameworks. An example of this is the FADE sequence frequently used within quality improvement circles (see E. Davis, 2000; Hunter & Love, 1996). The sequence is referred to as FADE—an acronym for each of the four steps: Focus, Analyze, Develop and Execute—and is represented as a cycle with ‘breaks’ between each of the four steps. These breaks suggest that people working with FADE are encouraged to return to an earlier step should they discover, for instance, that they need to better refine the problem or issue on which they are concentrating. So, it is with the Evaluation Process Model: while the process is cyclical in nature, it is possible for individuals to return to an earlier phase if necessary.


Figure 21. Evaluation Process Model.

The Evaluation Process Model is arranged in four phases, and within each phase are several steps or suggested activities for teachers and leaders. The movement is multi-directional with breaks between the phases to indicate that individuals can return to an earlier phase if necessary. The model is cyclical in nature and depicts the necessary two-way flow of information that informs every phase of the evaluation process.

In Print’s (1993) model of curriculum development, which informed the design of the Evaluation Process Model, three phases are identified: Phase 1 – Organisation, Phase 2 – Development and Phase 3 – Application. Phase 1 is necessary in order to ensure that all the arrangements for the subsequent curriculum work are in place. Phase 2 is where the curriculum is developed, beginning with a situational analysis, as mentioned above, and working through the other key components of curriculum, including the intentions, teaching and learning activities, and the assessment of students’ learning. In Phase 3, teachers are encouraged to monitor, modify, evaluate and reflect upon the curriculum, which then informs its next iteration, thereby ensuring the curriculum is subjected to a continuous improvement strategy.

Similarly, with the Evaluation Process Model, in the Focussing phase, the school leader works with teachers to define the purpose and processes of conducting an evaluation of teachers’ professional growth needs and ensures that the preconditions for conducting the evaluation are in place. This phase may also encompass a review of any existing systems and practices at the school to ensure that this is appropriate, and supports the newly established purpose. Initiating then takes place to establish the infrastructure and resources exist for a plan for implementing the process to be created. The Evaluating phase is where the process is implemented: information is gathered and collated and is then analysed collaboratively by the teacher and school leader following the agreed processes. Consequently, a judgement is made about the subject of the evaluation, such as teachers’ professional growth progress, new goals and targets are identified, and actions are considered. In the final phase, teachers and school leaders critically reflect upon not only the processes that were followed, but also on the decisions made. This feedback is considered and informs the need for modifications to processes, infrastructure and resourcing at the system level. To ensure that this process supports Supporting Learning, strategies are put in place to share the lessons learned in the final phase of the model.

Importantly, the model is built on the foundation of educational program evaluation standards: feasibility, utility, propriety and accuracy. The cycle should facilitate feedback and continuous refinement to enable teacher, school and system-level professional growth.


The Evaluation Process Model is applied in the following section using a scenario about Josefina, a hypothetical Professional Learning Team Leader working in a remote school in Australia.

6.2. Implementing the Evaluation Process ModelThis section outlines the Evaluation Process Model and is intended as an example structure that could be developed further developed to assist educators, including all system stakeholders, in prioritising, planning, evaluating and synthesising information about supporting teachers’ professional growth. Throughout the following sections, links to relevant and useful resources have been provided, and a scenario has been developed progressively to apply each of the steps outlined above to contextualise how this process of evaluation can be used. The scenario used to illustrate each of the steps in the evaluation process can be found in the table below.

Table 28. Evaluation Process Scenario

Scenario: Professional Learning Team leader, remote school

Josefina is a secondary teacher and inclusive education specialist, who has worked in a remote school for four years. The secondary school has a lower-than-average ICSEA of 923, with 39% of the student cohort identifying as Indigenous and 34% of students are from non-English speaking backgrounds. Josefina leads a Professional Learning Team (PLT) at the school. The PLT has been concentrating on the ways in which they seek and provide constructive feedback to each other. The staff members in the PLT represent a range of backgrounds and experiences, and they are gradually developing the ability to feel comfortable in seeking advice and constructive feedback.

However, challenges have arisen in seeking to translate their constructive feedback into relevant goals for their professional learning and measuring their progress towards these goals following their performance reviews. Josefina has an upcoming performance review and feels inexperienced and nervous at the prospect of managing self-evaluation and measuring her progress toward her goals. She requires support in ensuring her evaluative feedback can be translated into clear, measurable next steps for improvement within a framework that she can utilise to confidently measure and track her progress, and that of the PLT that she leads, across the year.

6.3. Focussing i. Analyse current situation

An analysis of the current situation needs to take place. Specifically, this should involve a situational or needs analysis, where the current state of progress about the identified focus is described, alongside any barriers that have been identified. In practical terms, this is often included as a part of the theory of action. Many program logic model templates include components of a situational analysis, which can be undertaken via a SWOT Analysis.

Documenting the theory of action is similar to using a graphic organiser so that there is a clear line of sight between the goals, inputs, activities and outcomes. This could be done in a variety of ways, such as through the use of a logic model, which is a conceptual representation of a theory of change (Department of Human Services, 2009). A simple introduction to logic modelling for an education context has been developed by the NSW Department of Education.

Scenario: Identify focus


After a shared evaluation purpose has been defined, Josefina and the Professional Learning Team meet to map out the sequential steps required in receiving feedback to set professional learning goals. The team identify classroom observations and peer feedback as inputs in their theory of action which lead to teacher reflection, setting professional learning goals, and finally implementation of new practices. The PLT conceptualise this process as a continuous cycle of feedback and learning, which will ultimately lead to improved teacher practice and improved student outcomes.

At the end of this phase, the focus of the evaluation should be established along with an identification of scope and readiness of the system to undertake the evaluation, hence planning for the evaluation can then occur.

ii. Establish pre-conditionsOnce the purpose and process of the evaluation have been identified, it is essential to assess the readiness of the system (which may be the school) to undertake an evaluation. In this study, necessary preconditions of a teacher evaluation system have been identified. This includes ensuring that time and resources are made available for the evaluation to take place, and ensuring that the school culture is respectful, trusting and supportive.

Scenario: Prioritise purpose and process

Josefina invites the principal and other school leaders to attend the next Professional Learning Team meeting so that they can collaboratively reflect on their readiness to undertake the evaluation. They consider the time and resources available and the potential usefulness of the evaluation and decide that while they are not sure that all of them have the available time, enough of them are in place to progress with planning for the evaluation.

iii. Prioritise purpose and processIrrespective of the size or scope of any evaluation project or activity, ensuring that the school is ready to conduct an evaluation and that there is buy-in from leadership and teachers are important steps. This ensures the evaluation is feasible, that a common professional growth goal can be established, and that findings and any new goals and suggested actions are followed.

At this stage of the evaluation process, determining the purpose, or in other words, the scope, of the evaluation activity is important. This should be a collaborative endeavour between the teacher(s) participating in the evaluation and school leaders.


Scenario: Prioritise purpose and process

At a Professional Learning Team meeting, Josefina raises the need for further support in translating constructive feedback into relevant goals for professional learning and performance reviews. The PLT members at the meeting agree with Josefina, that they would also like support translating constructive feedback. A separate meeting is set by the principal to determine if the necessary conditions to conduct an evaluation are present and confirm the purpose of the evaluation with teachers, middle leaders and other school leaders. There is a consensus that the purpose of the evaluation will be to evaluate the Professional Learning Team’s current methods of providing constructive feedback, identifying any gaps and assessing the appropriateness of other tools to supplement current feedback mechanisms.

6.4. Initiating iv. Consider available resources

Evaluation can be resource and time-consuming. In this step of evaluation planning, it is important to consider timelines, budget and availability of staff.

TimelinesDefining the evaluation timeline depends on several key factors, including:

1. Chosen methodology and data collection activities, eg Does the evaluation involve the review of potential teacher evaluation tools or more time-intensive activities such as teacher interviews to gauge perception of current tools and activities?

2. Timing of expected ‘outcomes’ or ‘activities’ being captured in the evaluation, eg Are classroom observations being conducted at the beginning or end of the school term?

3. The level of detail in reporting requirements, eg Does the school executive require a written document, or will findings and recommendations be presented orally at the staff meeting?

With respect to data collection activities, it is preferable to utilise activities that are already in place, such as teachers conducting peer observations of teaching. If it is decided that new forms of gathering evidence are necessary, such as utilising a new online tool for measuring teacher practice, developing a survey or conducting an interview, it is important to be mindful of the time required for implementation, or for developing these instruments: creating a new survey, for example, often takes time to develop, however once the survey is developed, the collection process requires much less time compared with interviews, where the opposite is true. An interview schedule takes less time to develop than a survey, but data gathering time for interviews is far greater than for an online survey.

Budget There should be a dedicated budget for evaluation, as different tools have different costs associated with them, mainly due to timing considerations. As examples, interviews can cost more than surveys because of the time required to collect and analyse data, and some online teacher evaluation tools can have added costs.

Staff availability Evaluation can be time consuming. It is important to consider how much time staff have to conduct and engage in evaluation, and whether they are given support to ensure that they are able to fulfil other responsibilities within the school. It is also possible, particularly in smaller or remote schools, that people with expertise external to the school may need to be sourced. Engaging consultants – whether from within the system or external to it – has some costs associated. A work chart (GANTT chart) might be useful for allocating tasks according to availability.


v. Establish or review infrastructure to enable processDepending on the focus of the evaluation and the level of system readiness, either mobilisation or establishment of infrastructures may be necessary prior to the evaluation being able to occur. For instance, infrastructure may include physical systems to store information. It is important that during the prior stage, particularly in the analysis of the current situation, that detailed examination of the necessary preconditions and infrastructures takes place. Further, it should be noted that ‘infrastructure’ also includes processes and policies that may influence the success of the evaluation, for instance, the degree to which professional collaboration is evident in the school culture:

…to successfully grow practice and sustain potential impact on increasing student achievement, Collaborative Learning among teachers requires an atmosphere in which system and school leaders are intentional in demonstrating that they are equally committed to continuous learning alongside teacher, students and each other. (Sharratt & Planche, 2016, p. 36)

In other words, the preconditions for supporting effective evaluation systems must be present across the school to enable the process and to engage in evaluation practices: these are shared clarity of purpose and explicit criteria, relational trust, supportive school leadership and climate, and an evaluative mindset.

Scenario: Infrastructure to support system

Through the Professional Learning Team, collaboration in providing feedback and the articulation of the shared understanding of the evaluation has occurred and has become a standing agenda item on upcoming team meetings, and that there is resourcing funds set aside by school leadership to prioritise this. If these factors are not present, the successful implementation of the evaluation may be hindered, and the school would first need to work on facilitating a collaborative learning culture and enabling adequate resourcing and infrastructures.

vi. Create plan for implementing resourcesOnce the available resources for the evaluation are determined, how the evaluation will be conducted, including what evaluation questions will be asked, needs to be determined. Evaluation questions focus the evaluation and ensure a specific question about the theory of action set out in the Initiate System phase can be answered. The methods used to answer these questions are then set out in an evaluation plan which will guide the implementation of the evaluation. A guide that assists teachers and school leaders develop key evaluation questions can be found on the NSW Department of Education evaluation resource hub .

In addition to evaluation questions and overall plan, the evaluation plan should also include the tools used to collect data. The data collection and analysis for evaluation reference guide for teachers provides a reference for common data collection methods including interviews and focus groups, surveys, observations and document analysis.

Scenario: Set up implementation process

Looking at their theory of change, and reflecting on the shared understanding of the purpose of the evaluation, Josefina and the PLT collectively decide the evaluation will focus on the following questions:

- To what extent does the PLT peer feedback facilitate teacher reflection and goal setting? - What additional resources are available to facilitate teacher reflection and goal setting?

In their evaluation plan, the PLT identifies the initial need to determine appropriate measures that would provide the necessary information to address the two questions. The PLT decide to use the Rubric for Appraising Evaluation Tools and Resources, and collectively assess the available resources against their most pertinent principles for teacher evaluation: that the information generated by the tool in relation to their reflection and goal setting processes promotes teacher professional growth, and that it can capture with authenticity the current practices of teachers. After having assessed the available resources, as well as a wider range of tools that are


accessible to the school, the PLT decides to use an anonymous online survey tool and interview protocol that had been developed by a Leading Teacher in the previous school year, agreeing that it will meet the needs of this evaluation, and that there are the available resources for implementation.

6.5. EvaluatingIn addition to a systematic process to determine the merit, worth and significance of the program or activity being evaluated, evaluation is also a continuous cycle of feedback underpinned by the evaluation standards, namely: feasibility, utility, propriety and accuracy. The implementation of the evaluation plan takes place in this feedback cycle and includes data collection, analysis of information, sharing lessons learned, resetting goals, and next steps.

vii. Implement process Implementing the plan for the evaluation is the first step. This often involves ensuring that all the infrastructure for the evaluation plan is in place prior to the gathering and collation of information. It may also involve developing data collection tools and setting up databases for ensuring that data can be readily analysed once it has been gathered.

viii. Gather and collate information This second step in the implementation of the evaluation is data collection guided by the evaluation plan and the tools developed during the planning stage of the evaluation process. Ideally, implementation of the evaluation plan should be according to the timeline and budget previously established, although some adjustments may be necessary. As is set out by the multi-directional process model, teachers and leaders can return to the earlier stages, for example to revisit the available resources and determine an alternative plan for implementation. This is explored in the below scenario.

Scenario: Implement system

It is currently the middle of Term 2, and looking at the school calendar, the PLT notes that the two-day athletics carnival and the delivery of a full day mental health workshop by an external group are upcoming, therefore affecting the availability of staff to set aside time to engage in the process, and timeliness of feedback on the efficacy of peer feedback in PLTs to facilitate reflection and goal setting processes.

The PLT decides to proceed with the established implementation process for evaluation, however agree to commence data collection in Term 3, as this will give them enough time to finalise the survey and interview protocol that will meet the identified purpose, and adhere to the principles of the evaluation process. Over the course of weeks 2-5 of Term 3, the PLT conducts observations and feedback sessions with 11 teachers. At the end of each feedback session teachers are asked to fill out an anonymous online survey. In addition to this survey, three teachers consent to participate in an in-depth interview.

ix. Analyse informationData analysis depends on the type of data collected. Having a data analysis plan before data collection will ensure that it is possible to appropriately answer the evaluation questions. This plan should include identifying the analysis technique for each data source being used and may also include the process for each analysis technique.

Often multiple sources of data are required to answer a posed evaluation question. Therefore, a triangulation process is sometimes necessary. Triangulation is the term used to ensure there is consistency in the findings obtained by using more than two sources of information. This helps validate the data and findings (Better Evaluation, 2018). The Victorian Department of Education and Training provides useful information about analysing and using data.

Scenario: Analysis of information


The PLT meets to decide how each data collection activity will be analysed. They use a table to describe the data collection method and the analysis technique.

Data collection method Analysis technique

Classroom observation Analysis of patterns eg in questioning techniques used

Interviews Thematic analysis ie Identify themes and code interview responses

Assessment of teacher evaluation tools

Use of rubric by three assessors to provide an overall assessment of available teacher evaluation tools

Using their analysis plan, the PLT analyses the classroom observation data, following the guidance materials provided by the Australian Institute for Teaching & School Leadership. They then consider the survey and the interview data. The survey and interview data show that teachers perceived the PLT peer observations and feedback as a good mechanism for identifying areas for teacher improvement. However, teachers also felt that the process was insufficient in facilitating teacher personal reflection, and therefore according to the theory of action unable to contribute goal setting and improved practices.

Based on these findings, the PLT revisits an earlier stage of the evaluation process, and appraises the available tools and resources that could supplement the current feedback system to facilitate teacher reflective practice and goal setting.

x. Develop a judgement based on informationAs schools continue to evaluate programs and activities and share lessons learned, they will be able to collate data and form an understanding of overall school program implementation or outcomes.

xi. Identify new goals and targetsEvaluation should be a continuous cycle of consultation and feedback. Once lessons have been shared, the goals and targets of the evaluation should be reviewed and reset. If the original evaluation was an


Scenario: Collate information and form a pattern

At the completion of the evaluation of the PLT observation and feedback, the PLT, in collaboration with school leaders, undertakes an evaluation of the extent to which curriculum leaders are able to support teachers to use formative assessment to monitor student learning progress, and differentiate teaching practices. Another purpose of the evaluation, which is shared by members of the PLT and curriculum leaders, is to understand how teachers are engaging in reflection and goal setting, specifically in relation to their formative assessment practices, at this later timepoint.

For this evaluation, they review the system for evaluation, implement the evaluation, and analyse the gathered information. Findings from this evaluation suggest that there is a number of practices occurring at the school that are leading to high-quality use of formative assessment, however, the process also highlights some areas for improvement, and the evaluative information that is generated provides some suggested aspects for curriculum leaders to focus on. The evaluation findings also suggest that the tools currently being used by curriculum leaders and teachers are not adequately enabling staff to monitor progress, determine the effectiveness of their current formative assessment practices, or providing the necessary information that informs adapted practice.

Collating the findings of this evaluation, and from the evaluative information gathered from PLT observations and feedback, the PLT identifies a common pattern: that teacher reflection processes need to be supported, and that there is a gap in the available tools and resources that can be used to engage in this process. From this, the PLT conducts a needs assessment to identify the types of resources teachers need at their school, and assesses these using a rubric.

implementation evaluation with the goal of adapting and improving a program or activity, the reset goals and targets may look at measuring whether these adaptations have meet expectations. If, however, the original evaluation was an impact evaluation looking at the outcomes of a program or activity, the school may decide to set new goals and targets.

xii. Consider next actions The next steps in the evaluation are determined by the goals and targets that have been reset. Next actions may be another iteration of data collection or beginning the evaluation process from the beginning by identifying and defining a new priority through a theory of action.

6.6. Supporting Learning Finally, System Learning is the fourth stage in the Evaluation Process Model. The ‘system’ could be a group of teachers, a school, a network of schools, an education region, a state or territory or the whole Australian education system. In this stage, system learning is necessary to act on the findings of the evaluation to continue to support the professional growth of teachers and of school leaders.

xiii. Critically reflect on decisions & decision-making processWhile this process may also occur in the Evaluation stage, some critical reflection on the judgement made within the scope of the evaluation is required to determine how it will/should affect decisions about aspects of the system that may influence future evaluations. For instance, it may be determined that the school leader needs to be present at more of the PLT meetings to ensure the system of observation and feedback is working and is prioritised over other components of teaching practice that tend to dominate the PLT meetings.

xiv. Modify system to support educator growthThe output of the critical reflection should be an indication of what, if any, modifications to the system are required to support teachers’ and school leaders’ professional growth. While these modifications may be considerable or minor, it is important to note that the critical reflection will help to ensure that suggested modifications are evidence-informed as reflection allows for professional judgement to be incorporated in the process of acting of evaluation information.

xv. Share lessons learnedWhile continuous feedback should occur throughout the evaluation process, there also needs to be a commitment to share lessons learned and to disseminate the findings of the evaluation to support system learning and to promote continued professional growth. To help to ensure that dissemination is effective, it is important to consider the audience and their information needs. For example, do the school executive, principal or teachers require the findings in different formats, when do they require the findings to make any decisions, and what information will help them make these decisions?

Scenario: Share lessons learned

Throughout the evaluation, the PLT found that the current system of observation and feedback was perceived favourably by teachers but was limited in its ability to facilitate teacher reflection. Through the evaluation, the PLT also identified a tool that can be used by mentors to facilitate teacher reflection and goal setting and assessed its suitability against a rubric. At a staff meeting Josefina and other members of the PLT present the findings of the


Scenario: Reset goals and targets, Consider next actions

After one term of implementing the new model of teacher observation and feedback, the PLT decides that it is important to collect further information on this process, to understand the extent to which the new model of PLT peer feedback facilitates teacher reflection and goal setting. To do this the PLT, guided by Josefina, redistributes the survey to measure outcomes and compare this to the first round of the survey.

evaluation and their plan to incorporate this new tool into teacher observation and feedback sessions, to better support teachers to reflect on their practices and help them set goals with measurable steps for improvement.

Monitoring and evaluation should be built into all programs and activities. Schools should consider how best to incorporate these activities into their overall school strategy to support continued professional growth.

This section has briefly described an application of the Evaluation Process Model. Testing and applying this model in practice would assist in further developing more detailed and bespoke resources for each stage. The following section details one resource that could be used in the Focussing and Initiating phases to select appropriate and fit-for-purpose evaluation tools or resources.


6.7. Rubric for Appraising Evaluation Tools and ResourcesBased on the findings of the study, a rubric was developed to assist educators in choosing appropriate and effective evaluation tools and resources. The rubric is developmental in nature, allowing users to assess how appropriate an evaluation tool is while considering the professional development needs of the relevant educators.

6.7.1.MethodA rubric for assessing evaluation tools was created by utilising the list of principles generated through multiple sources, including the literature scan that was tested through the Delphi survey rounds, and stakeholder workshops, where input from those activities refined and prioritised the principles. These formed the 12 rubric standards, which are rated on a four-point scale. Descriptors (or criteria) for the four-point scales were created through several testing processes, with the continuum of practices (Emerging, Evolving, Embedding and Excelling) reflecting the levels used in the Framework for Improving Student Outcomes (FISO) (Victoria State Government, 2019).

During its development, the rubric was tested and trialled in a two-part process:

1. The first part of this development process took place in a collaborative workshop with DET and AITSL. Workshop participants were provided with a description of an evaluation tool used in Australian contexts and a hypothetical school scenario in which the evaluation was to take place. Participants were asked to use a draft version of the rubric to make a rating against each of the principles as part of assessing the appropriateness of the evaluation tool. For this activity, only the end-point descriptors were provided, with participants asked to provide justifications for their assessment, and contribute to the development of the descriptors by providing suggestions, feedback, and developmentally-appropriate statement within the continuum. Following this workshop activity, the scale descriptors were created and updated with the received feedback informing this process.

2. The second part of this rubric development process involved another iteration of internal testing. An internal workshop was held to verify the language and appropriateness of descriptors for providing a judgement on an evaluation tool and school context. A similar procedure was undertaken during this stage, where participants were also asked to justify the ratings provided and share suggestions to add clarity to the descriptors.

It should be noted that while the rubric has been developed in collaboration with DET and AITSL stakeholders, in its current form, it will need to be tested for wider applicability and usability. This may involve testing the rubric with different users, in a range of contexts, and with a more extensive variety of evaluation tools and resources (particularly those that differ in method, for example across tools that use teacher self-reflection, teacher interviews, or peer collaboration etc.).

6.7.2. Intended UseThe rubric is situated within the Evaluation Process Model in the Focussing and Initiating phases. It is recommended that discussions around tool selection, and use of the rubric, should occur only after the first phase has taken place. The selection of an appropriate tool(s) (and therefore use of a practical rubric to inform selection) should occur during the second phase where the infrastructure, available resources, and plans for implementation are considered.

During the first phase of the evaluation process, before selecting a tool, professional discussion(s) should occur between the parties involved to determine a shared understanding and purpose. The intention is that this conversation should allow the parties to reach an agreement on the focus of the evaluation, the current situation, and therefore the essential content that the tool should capture and the principles that the tool should meet. Following on from this agreement, the rubric can be used to assess if a tool or


resource captures the critical content relevant to the teacher’s needs, provides the necessary information to contribute to the intended purpose of the evaluation, and that it meets the agreed-upon principles to a desired extent.

It is not essential that a tool or resource meets every principle or achieves the highest grading on each developmental continuum. The goal of the rubric is not to simply provide a summative ‘rating’ of the tool. Rather, it aims to assist teachers and school leaders to generate information about the tool’s fitness for purpose, that is, the capacity of the tool or resource to be used in a meaningful way by the relevant parties. An additional objective of this resource is to assist users in identifying strengths and weaknesses of a tool or resource, allowing for further consideration of the suitability of the tool or resource for the context or purpose of the evaluation, outside of any pre-determined principles. The decision around the appropriateness of the tool or resource should be made by the rubric users based on the how well it meets the needs and relevant principles, as previously identified through the professional conversation(s). This will ultimately inform the selection of an evaluation tool or resource that are fit-for-purpose and which meet the identified need and scope of the evaluation.

Tool or resource selection, as is situated within the Evaluation Process Model, is intended to be a cyclical process. It is recommended that users periodically reassess the selected evaluation tool(s) or resource(s) for appropriateness, in order to appraise whether it is providing the necessary evaluative information or whether it still holds relevance within its context, which may be transient or prone to change. In fact, iterative reappraisal of evaluation tools is a necessary consideration in order to be relevant and responsive to changing needs of educators.


Table 29. Rubric for Appraising Evaluation Tools and Resources

Principle Emerging (1) Evolving (2) Embedding (3) Excelling (4)

Authentic

Authentic professional growth assessments (inclusive of broader evaluation practices) should be undertaken within the context or setting where professional practice occurs to capture evidence of relevant skills, attitudes and knowledge within real-life practice context.

Assessment captures some relevant skills, attitudes and knowledge. Links to the practice context may be unclear.

Assessment captures skills, attitudes and knowledge which are reflective of real-life practice context.

Assessment occurs in real professional practice contexts and captures evidence of relevant skills, attitudes and knowledge.

Assessment occurs in real professional practice contexts and captures evidence of highly relevant skills, attitudes and knowledge.

Evidence-based (Assessor Expertise)

The professional growth assessment is carried out by educators who have comprehensive knowledge of and relevant expertise in the skills being assessed, and an understanding of the expected standards and performance assessment based on evidence as proof of competence when matched against the relevant APST(s).

Assessor has knowledge of the skills being assessed and uses evidence as proof of competence in these skills.

Assessor has knowledge of and expertise in the skills being assessed. They use evidence as proof of competence in these skills.

Assessor has knowledge and expertise in the skills being assessed. They understand the expected standards and use evidence to assess the teacher against the relevant standards (APST).

Assessor has comprehensive knowledge and considerable expertise in the skills being assessed. They understand the expected standards and use a breadth of evidence to assess teacher against the relevant standards (APST).

Collaborative

Implementing the professional growth assessments should involve high levels of communication, opportunities for reflection and contextual interpretation of information developed through using the tool.

Assessment allows for consultation or collaboration between the person being assessed and the assessor/s (eg during formative and/or summative stages, for reflection, or interpretation of data).

Assessment encourages collaboration between the person being assessed and the assessor (eg during formative and/or summative stages, for reflection, or interpretation of data).

Assessment requires collaboration between the person being assessed and assessor/s at some stages. It may also involve collaboration with peers and colleagues. (eg during formative and/or summative stages, for reflection, or interpretation of data)

Assessment integrates collaboration between the person being assessed and assessor/s throughout all stages of the assessment. Collaboration also occurs with peers and colleagues.

Context-specific

Professional growth assessments can be tailored to practice settings, individual school priorities and school context.

The assessment tool can be applied to the practice settings, individual school priorities or school context. The tool may

The assessment tool provides opportunities for meaningful adaptation to the practice settings, individual school

The assessment tool can be easily adapted to practice settings, individual school priorities and contexts.

The assessment tool is designed to be easily tailored to relevant practice settings, individual school priorities and



provide some opportunities for basic adaptation.

priorities and school context. contexts.

Comprehensive

Professional growth assessments use multiple modalities, multiple samples of evidence, or multiple assessors as evidence of competence.

Evidence of competence is based on a singular mode, sample of evidence or assessor.

Multiple modes, samples of evidence, OR assessors are used to provide evidence needed for judgement or evidence of competence.

A variety of modes, samples of evidence, OR assessors are used to provide evidence needed for judgement of competence (>3 per category).

A variety of modes, samples of evidence, OR assessors are used to provide evidence needed for judgement of competence (>3 per category). The selection of these aspects is appropriate and justified.

Understandable

Professional growth assessment domains, purpose, standards (if included), and feedback (or output) must be clear and easy to understand for all parties.

Assessment purpose(s) OR domain(s) OR feedback/output are defined.

Assessment purpose(s), domain(s) OR feedback/output are clear but may require further clarification for some of the relevant parties (eg the format of the feedback means that the assessor may need to clarify the results with the person being assessed).

Assessment purpose(s), domain(s) and feedback/output are clear and easy for all parties to understand.

Data/feedback/output provided by the assessment tool is in a format that is actionable and can inform next steps (eg for teacher practice).

Relevant

Professional growth assessments are linked to a relevant and diverse set of standards, competencies or performance metrics that are relevant to Australian teachers, school leaders and practice contexts.

Assessment is linked to a set of relevant standards, competencies or performance metrics.

Assessment is relevant for Australian teachers and practice contexts. There are some links between the tool and APST/APSP.

Assessment is relevant for Australian teachers and practice contexts. There are clear links between the tool and a diverse set of standards within the APST/APSP.

Assessment is highly relevant for Australian teachers and their practice contexts. The assessment is explicitly mapped to a diverse set of standards within the APST/APSP.

Professional growth

Professional growth assessment helps determine and prioritise areas for professional growth,

Assessment tool feedback/output can be used to identify areas for professional

Assessment tool assists in identifying areas for professional growth, learning

Assessment tool feedback/output clearly and easily supports the end-user to

Assessment tool feedback/output provides actionable feedback for users



identify professional learning opportunities and planning for career development via the provision of actionable feedback.

growth, learning opportunities or career development.

opportunities or career development.

identify areas for professional growth, learning opportunities or career development.

to identify areas for professional growth, learning opportunities or career development.

Regular

Assessing competence through professional growth assessment requires an accumulation of data or evidence about performance over a period of time to provide ongoing progress monitoring and feedback.

Data are collected to evidence performance at various points in time.

Data are collected repeatedly to evidence performance over a period of time.

Data are collected regularly and provides ongoing progress monitoring/feedback.

Data are collected regularly and provides ongoing progress monitoring/feedback on a variety of specific practices.

Student-based

Student-based professional growth assessments involve students in the feedback/evaluation process, measurement of growth or entail observation of classroom interactions.

Assessment involves student input either directly or indirectly.

Assessment takes student input into consideration.

Assessment involves appropriate and useful student input.

Assessment involves high levels of appropriate and useful student input.

Professional Agency

Professional growth assessments and domains examined should be self-directed by the teacher based on self-identified strengths, areas of development, and interests, rather than being compliance-based or solely driven by other school improvement agendas.

Assessment tool allows for some consideration of individual needs/interests (eg in implementation or use of feedback/outputs).

Assessment tool allows for substantial consideration of individual needs/interests (eg in implementation or use of feedback/outputs).

Assessment tool can be practitioner driven (e.g. implemented by the assessed teacher) and considers individual needs/ interests (eg in implementation or use of feedback/outputs).

Selection, implementation and feedback/outputs of the tool reflect the teacher’s self-identified strengths, areas of development, or interests. It allows the teacher to take ownership of their professional development journey.

Utility

Professional growth assessments should be tailored to meet identified areas of professional growth and learning and inform growth opportunities for career development, and/or as aligned with school improvement objectives.

The assessment tool generates output which has some alignment with the identified areas for professional growth and/or with school improvement objectives.

The assessment tool generates appropriate output which is aligned with identified areas for professional growth and/or school improvement

The assessment tool generates appropriate output which can be used to inform teacher’s professional development journey and/or contributes to achievement of school

The assessment tool generates feedback/output that is highly fit for purpose and gives direction for teacher’s professional development journey or



objectives. improvement objectives. contribute significantly to the achievement of school improvement objectives.


7.Closing CommentsPart B of the Teacher Practice Evaluation Framework study has provided and described two resources (referred as the Framework) to support the development and enhancement of evaluation systems that support teachers’ professional growth, and evaluation practices that are underpinned by appropriate educational evaluation standards and the principles of effective teacher evaluation as defined and verified by the more than 150 stakeholders from the Australian education system who participated in this study. It is acknowledged that evaluation is occurring in many ways across Australian schools, it is envisaged that this Framework could support and enhance this work.

As referred to earlier in this part, more testing and refinement through application of the Framework is necessary to develop bespoke resources and case examples to support widespread uptake and refinement of the Evaluation Process Model and rubric for appraising evaluation tools and resources.

It is our view that the most effective way to do this will be to partner with those educators working in schools, and importantly state and territory education departments, regulatory authorities, education unions and professional associations. The widespread participation of stakeholders from different parts of the system demonstrates the appetite for effective teacher evaluation in Australian, and further, have


Case Profile

The following profile highlights how the rubric can be used within a school context.

• Jon is a secondary school teacher who is in his third year of teaching. In the past, he has used positive behaviour management strategies to create safe and productive learning environments for his students. However, at the start of this year, Jon found himself teaching a Year 10 mathematics class with a great number of disengaged students. As a result of this disengagement, many students exhibit challenging behaviours, making it difficult for Jon to teach the content effectively.

• As part of his professional development plan, Jon decides to meet with the assistant principal to set a course for his development across the year. Based on the assistant principal’s observation of Jon’s classes in previous years, the pair establishes that Jon has a well-developed understanding of positive behaviour management strategies. To combat the disengagement, the assistant principal suggests that Jon could focus instead on supporting student participation within his mathematics classroom. The pair discuss potential ways to collect evidence about Jon’s performance in this area, and how this can later be used to provide actionable steps for his professional development.

• Of the range of tools that are currently used by the school, Jon and the assistant principal decide that it is important to choose one with specific properties that would measure certain practices, and provide information that could be acted upon, and to meet Jon’s identified goal. Due to the nature of the identified goal, Jon decides that it is important that the selected evaluation tool is student-based, and that it is used by a trusted colleague at the school who has expertise in strategies for supporting student participation.

• The rubric was utilised as a checklist to assess whether any of the tools available at the school would meet this goal. They assessed two tools against the rubric and found that both met these principles to a good extent. However, one of the tools was found to be more comprehensive, as it encompassed a variety of modes of data collection. Based on judgement against the rubric, he decided that this tool would provide a more holistic depiction of his practice. It was therefore selected as it supported Jon’s professional development needs to a greater extent. Jon and his assistant principal agreed to set up another meeting to discuss next steps and to identify where the school could provide support for Jon to engage in use of this tool.

ensured the Framework is based upon a variety of views about teaching and the provision of quality education. Continuing to engage in wide consultation in the testing of the Framework would only serve to further maximise its applicability across the diversity of educational contexts in Australia.

8.ReferencesACER. (2017). Principles of Good Practice in Learning Assessment. Melbourne: ACER.

Australian Association of Social Workers. (2019). Creating a CPD Plan. Retrieved from https://www.aasw.asn.au/professional-development/creating-a-cpd-plan

Australian Institute for Teaching and School Leadership. (2014). Australian Professional Standard for Principals and the Leadership Profiles. Carlton South, VIC: Education Services Victoria.

Australian Institute for Teaching and School Leadership. (2017). Guide to the certification of highly accomplished and lead teachers in Australia. Retrieved from https://www.aitsl.edu.au/tools-resources/resource/guide-to-the-certification-of-highly-accomplished-and-lead-teachers-in-australia

Australian Institute for Teaching and School Leadership. (2018a). Australian Professional Standards for Teachers. Retrieved from https://www.aitsl.edu.au/teach/standards

Australian Institute for Teaching and School Leadership. (2018b). Australian Teacher Performance and Development Framework. Retrieved from https://www.aitsl.edu.au/docs/default-source/national-policy-framework/australian-teacher-performance-and-development-framework.pdf?sfvrsn=4a7fff3c_6

Better Evaluation. (2018). Trangulation. Retrieved from https://www.betterevaluation.org/en/evaluation-options/triangulation

Better Evaluation, (2019). Strengthen evaluation capacity. Retrieved from https://www.betterevaluation.org/en/rainbow_framework/manage/strengthen_evaluation_capacity

Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability (formerly: Journal of Personnel Evaluation in Education), 21(1), 5.

Brady, L., & Kennedy, K. (2013). Curriculum construction: Pearson Higher Education AU.

Brookhart, S. M. (2011). Educational assessment knowledge and skills for teachers. Educational Measurement: Issues and Practice, 30(1), 3-12.

Buckley, J., Archibald, T., Hargraves, M., & Trochim, W. M. (2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American journal of evaluation, 36(3), 375-388.

Buja, A., Swayne, D. F., Littman, M., Dean, N., Hofmann, H., & Chen, L. (2008). Data visualization with multidimensional scaling. Journal of Computational and Graphical Statistics, 17, 444-472.

Camburn, E. M., & Han, S. W. J. J. o. E. C. (2015). Infrastructure for teacher reflection and instructional change: An exploratory study. 16(4), 511-533.

Centre for Education Statistics and Evaluation. (2016). 5 essentials for effective evaluation. Retrieved from https://www.cese.nsw.gov.au/publications-filter/5-essentials-for-effective-evaluation

Charteris, J., & Smardon, D. (2019). The politics of student voice: unravelling the multiple discourses articulated in schools. Cambridge Journal of Education, 49(1), 93-110.


https://www.cese.nsw.gov.au/publications-filter/5-essentials-for-effective-evaluation

https://www.betterevaluation.org/en/rainbow_framework/manage/strengthen_evaluation_capacity

https://www.betterevaluation.org/en/rainbow_framework/manage/strengthen_evaluation_capacity

https://www.betterevaluation.org/en/evaluation-options/triangulation

https://www.betterevaluation.org/en/evaluation-options/triangulation

https://www.aitsl.edu.au/docs/default-source/national-policy-framework/australian-teacher-performance-and-development-framework.pdf?sfvrsn=4a7fff3c_6



https://www.aitsl.edu.au/teach/standards

https://www.aitsl.edu.au/tools-resources/resource/guide-to-the-certification-of-highly-accomplished-and-lead-teachers-in-australia



https://www.aasw.asn.au/professional-development/creating-a-cpd-plan

Clinton, J., Anderson, M., Dawson, G., Dawson, A., Bolton, S., & Mason, R. (2016). Teacher Effectiveness Systems, Frameworks and Measures: A Review. In: Melbourne: Centre for Program Evaluation. Find this resource.

Clinton, J., Aston, R., & Koelle, M. (2018). Investigating the key determinants of effective teaching: a systematic review. In: Reported prepared for the Australian Government Department of Education and ….

Clinton, J., & Dawson, G. (2018). Enfranchising the profession through evaluation: a story from Australia. Teachers and Teaching, 24(3), 312-327.

Clinton, J., Dinham, S., Savage, G., Aston, R., Dabrowski, A., Gullickson, A., . . . Arbour, G. (2015). Evaluation of the Implementation of the Australian Professional Standards for Teachers.

Close, K., Amrein-Beardsley, A., & Collins, C. (2018). State-Level Assessments and Teacher Evaluation Systems after the Passage of the Every Student Succeeds Act: Some Steps in the Right Direction. Retrieved from

Cohen, J., McCabe, L., Michelli, N. M., & Pickeral, T. (2009). School climate: Research, policy, practice, and teacher education. Teachers college record, 111(1), 180-213.

Cosner, S. (2009). Building organizational capacity through trust. Educational Administration Quarterly, 45(2), 248-291.

Creswell, J. W., & Poth, C. N. (2017). Qualitative inquiry and research design: Choosing among five approaches: Sage.

Crossley, J., & Jolly, B. (2012). Making sense of work based assessment: ask the right questions, in the right ‐way, about the right things, of the right people. Medical education, 46(1), 28-37.

Danielson, C. (2011). Evaluations that help teachers learn. Educational leadership, 68(4), 35-39.

Danielson, C., & McGreal, T. L. (2001). New trends in teacher evaluation. Evaluating Educators, 58(5), 12-15.

Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of state policy evidence. Education policy analysis archives, 8, 1.

Darling-Hammond, L. (2012). Powerful Teacher Education.: Lessons from Exemplary Programs: John Wiley & Sons.

Darling-Hammond, L., Hyler, M. E., & Gardner, M. (2017). Effective teacher professional development. In: Palo Alto, CA: Learning Policy Institute.

Darling-Hammond, L., & McLaughlin, M. W. (1995). Policies that support professional development in an era of reform. Phi delta kappan, 76(8), 597-604.

Darling-Hammond, L., & McLaughlin, M. W. (2011). Policies that support professional development in an era of reform. Phi Delta Kappan, 92(6), 81-92.

Darling-Hammond, L., & Richardson, N. (2009). Teacher learning: What matters? How teachers learn. Journal of Teacher Education, 66(5), 46-45.

Davis, D. R., Ellett, C. D., & Annunziata, J. (2002). Teacher evaluation, leadership and learning organizations. Journal of personnel evaluation in education, 16(4), 297-301.

Davis, E. (2000). A Quality Improvement Project in Diabetes Patient Education During Hospitalization. Diabetes Spectrum, 13(4), 234.


Department of Health and Human Services. (2019). Credentialing, competency and capability framework. Retrieved from https://www2.health.vic.gov.au/health-workforce/allied-health-workforce/allied-health-ccc-framework

Department of Human Services. (2009). Understanding program logic. Victoria State Government Retrieved from http://docs2.health.vic.gov.au/docs/doc/EF8861765B99DB1ECA257B19007DBA75/$FILE/understanding_program_logic.pdf.

Department of Human Services. (2012). Capability review: Department of Human Services. Retrieved from https://www.apsc.gov.au/sites/default/files/1217-DHS_capability_review.pdf

Derrington, M. L., & Campbell, J. W. (2015). Implementing new teacher evaluation systems: Principals’ concerns and supervisor support. Journal of Educational Change, 16(3), 305-326.

DiPaola, M. F. H., W.K.,. (2014). Improving instruction through supervision, evaluation, and professional development. Charlotte, NC: Information Age Publishing.

Dugard, P., Todman, J. B., & Staines, H. (2010). Approaching multivariate analysis: A practical introduction: Routledge.

Earl, L., & Timperley, H. (2015). Evaluative thinking for successful educational innovation.

Filipe, M. N. M., Ferreira, F. A. F., & Santos, S. P. (2015). A multiple criteria information system for pedagogical evaluation and professional development of teachers. Journal of the Operational Research Society, 66(11), 1769-1782.

Fleming, D. (2017). Student voice: An emerging discourse in Irish education policy. International Electronic Journal of Elementary Education, 8(2), 223-242.

Frase, L. E., & Streshly, W. (1994). Lack of accuracy, feedback, and commitment in teacher evaluation. Journal of personnel evaluation in education, 8(1), 47-57.

Freeman, C., O’Malley, K., & Eveleigh, F. (2014). Australian teachers and the learning environment: An analysis of teacher response to TALIS 2013: Final Report. Melbourne: ACER, 204.

Gamer, M., Lemon, J., Fellows, I., & Singh, P. (2013). Package irr: Various Coefficients of Interrater Reliability and Agreement (Version 0.84)[software]. In.

Goe, L., Biggers, K., & Croft, A. (2012). Linking Teacher Evaluation to Professional Development: Focusing on Improving Teaching and Learning. Research & Policy Brief. National Comprehensive Center for Teacher Quality.

Govaerts, M., & van der Vleuten, C. P. (2013). Validity in work based assessment: expanding our horizons. ‐Medical education, 47(12), 1164-1174.

Hair, J. F. J., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate Data Analysis (5th ed.). Upper Saddle River, New Jersey: Prentice Hall.

Hallinger, P. (2011). Leadership for learning: Lessons from 40 years of empirical research. Journal of Educational Administration, 49(2), 125-142.

Hallinger, P., Heck, R. H., & Murphy, J. (2014). Teacher evaluation and school improvement: An analysis of the evidence. Educational Assessment, Evaluation and Accountability, 26(1), 5-28.

Hallinger, P., & Lu, J. (2014). Modelling the effects of principal leadership and school capacity on teacher professional learning in Hong Kong primary schools. School Leadership & Management, 34(5), 481-501.


https://www.apsc.gov.au/sites/default/files/1217-DHS_capability_review.pdf

http://docs2.health.vic.gov.au/docs/doc/EF8861765B99DB1ECA257B19007DBA75/$FILE/understanding_program_logic.pdf

http://docs2.health.vic.gov.au/docs/doc/EF8861765B99DB1ECA257B19007DBA75/$FILE/understanding_program_logic.pdf

https://www2.health.vic.gov.au/health-workforce/allied-health-workforce/allied-health-ccc-framework

https://www2.health.vic.gov.au/health-workforce/allied-health-workforce/allied-health-ccc-framework

Hattie, J. (2008). Visible learning: A synthesis of over 800 meta-analyses relating to achievement: routledge.

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational research, 77(1), 81-112.

Hattie, J., & Zierer, K. (2017). 10 Mindframes for Visible Learning:Teaching for Success. Routledge.

Hill, H., & Herlihy, C. (2011). Prioritizing teaching quality in a new system of teacher evaluation. Education Outlook, 9, 1-6.

Holmboe, E. S. (2017). Work-based Assessment and Co-production in Postgraduate Medical Training. GMS journal for medical education, 34(5).

Hunter, M. E., & Love, C. C. (1996). Total quality management and the reduction of inpatient violence and costs in a forensic psychiatric hospital. Psychiatric services (Washington, DC), 47(7), 751-754.

Jensen, B. (2010). The OECD Teaching and Learning International Survey (TALIS) and teacher education for diversity.

Kane, T., & Steiger, D. (2012). MET Project: Gathering Feedback for Teaching. Retrieved from

Kania, J. & Kramer, M. (2011). Collective Impact. Stanford Social Innovation Review, retrieved from https://ssir.org/articles/entry/collective_impact

Keamy, R. (2016). Creative leadership?‘It’s just the norm’. School Leadership & Management, 36(2), 151-168.

Kelly, K. O., Ang, S. Y. A., Chong, W. L., & Hu, W. S. (2008). Teacher appraisal and its outcomes in Singapore primary schools. Journal of Educational Administration, 46(1), 39-54.

Kennedy, M. (2010). Teacher assessment and the quest for teacher quality: A handbook: John Wiley & Sons.

Kruskal, J. B. (1964). Nonmetric multidimensional scaling: a numerical method. Psychometrika, 29(2), 115-129.

Kyriakides, L., & Demetriou, D. (2007). Introducing a teacher evaluation system based on teacher effectiveness research: An investigation of stakeholders’ perceptions. Journal of personnel evaluation in education, 20(1-2), 43-64.

Le Fevre, D. M., & Robinson, V. M. (2015). The interpersonal challenges of instructional leadership: Principals’ effectiveness in conversations about performance issues. Educational Administration Quarterly, 51(1), 58-95.

Le Fevre, D. M., Robinson, V. M., & Sinnema, C. E. (2015). Genuine inquiry: Widely espoused yet rarely enacted. Educational Management Administration & Leadership, 43(6), 883-899.

Li, L., Hallinger, P., & Walker, A. (2016). Exploring the mediating effects of trust on principal leadership and teacher professional learning in Hong Kong primary schools. Educational Management Administration & Leadership, 44(1), 20-42.

Liu, S., Hallinger, P., & Feng, D. (2016). Supporting the professional learning of teachers in China: Does principal leadership make a difference? Teaching and teacher education, 59, 79-91.

Lortie, D. C. (2009). School principal: Managing in public: University of Chicago Press.

Marzano, R. J., Frontier, T., & Livingston, D. (2011). Effective supervision: Supporting the art and science of teaching.: ASCD.

Matter, T. (2009). Evaluating and rewarding the quality of teachers: International practices. Retrieved from


https://ssir.org/articles/entry/collective_impact

Muñoz, M. A., & Chang, F. C. (2007). The elusive relationship between teacher characteristics and student academic growth: A longitudinal multilevel model for change. Journal of personnel evaluation in education, 20(3-4), 147-164.

National Education Framework Cancer Nursing. (2019). Assessment Fact Sheet: Performance Assessment using Competency Assessment Tools. Retrieved from http://edcan.org.au/assets/edcan/files/docs/EdCan-FactSheet-CATs.pdf

Nursing and Midwifery Board of Australia. (2019). Fact sheet: Continuing professional development. Retrieved from https://www.nursingmidwiferyboard.gov.au/Codes-Guidelines-Statements/FAQ/CPD-FAQ-for-nurses-and-midwives.aspx

Odden, A. R. (2011). Strategic management of human capital in education: Improving instructional practice and student learning in schools: Routledge.

OECD. (2013). Teachers for the 21st Century – Using Evaluation to Improve Teaching. Retrieved from www.oecd.org/site/eduistp13/TS2013%20Background%20Report.pdf

Ofsted. (2015). The common inspection framework: Education, skills and early years. Office for Standards in Education.

Opfer, V. D., Pedder, D. J., & Lavicza, Z. (2011). The influence of school orientation to learning on teachers' professional learning change. School Effectiveness and School Improvement, 22(2), 193-214.

Patton, K., Parker, M., & Pratt, E. (2013). Meaningful learning in professional development: Teaching without telling. Journal of Teaching in Physical Education, 32(4), 441-459.

Patton, K., Parker, M., & Tannehill, D. (2015). Helping teachers help themselves: Professional development that makes a difference. NASSP bulletin, 99(1), 26-42.

Peña-López, I. (2009). Creating effective teaching and learning environments: First results from TALIS. Retrieved from

Print, M. (1993). Curriculum Development and Design. Sydney: Allen and Unwin.

Reinhorn, S. K., Johnson, S. M., & Simon, N. S. (2017). Investing in development: Six high-performing, high-poverty schools implement the Massachusetts teacher evaluation policy. Educational Evaluation and Policy Analysis, 39(3), 383-406.

Ripley, B., Venables, B., Bates, D. M., Hornik, K., Gebhardt, A., Firth, D., & Ripley, M. B. (2013). Package ‘mass’. Retrieved from https://cran.revolutionanalytics.com/web/packages/MASS/MASS.pdf

Ritter, G. W., & Barnett, J. H. (2016). Learning on the job: Teacher evaluation can foster real growth. Phi Delta Kappan, 97(7), 48-52.

Robinson, V., Hohepa, M., & Lloyd, C. (2009). School leadership and student outcomes: ldentififing what works and why: Best evidence synthesis iteration. Wellington, New Zealand: Ministry of Education, 29, 2011.

Sammon, J. W. (1969). A nonlinear mapping for data structure analysis. IEEE Transactions on computers, 100(5), 401-409.

Schreier, M. (2012). Qualitative content analysis in practice: Sage Publications.

Selkrig, M., & Keamy, K. (2015). Promoting a willingness to wonder: moving from congenial to collegial conversations that encourage deep and critical reflection for teacher educators. Teachers and Teaching, 21(4), 421-436.


https://cran.revolutionanalytics.com/web/packages/MASS/MASS.pdf

https://www.nursingmidwiferyboard.gov.au/Codes-Guidelines-Statements/FAQ/CPD-FAQ-for-nurses-and-midwives.aspx

https://www.nursingmidwiferyboard.gov.au/Codes-Guidelines-Statements/FAQ/CPD-FAQ-for-nurses-and-midwives.aspx

http://edcan.org.au/assets/edcan/files/docs/EdCan-FactSheet-CATs.pdf

Sharratt, L., & Planche, B. (2016). Leading Collaborative Learning: Empowering Excellence: Corwin Press.

Sinnema, C. E., & Robinson, V. M. (2007). The leadership of teaching and learning: Implications for teacher evaluation. Leadership and policy in schools, 6(4), 319-343.

Slater, H., Davies, N. M., & Burgess, S. (2012). Do teachers matter? Measuring the variation in teacher effectiveness in England. Oxford Bulletin of Economics and Statistics, 74(5), 629-645.

Stronge, J. H. (2005). Evaluating teaching: A guide to current thinking and best practice: Corwin Press.

Stronge, J. H., & Tucker, P. D. (2003). Handbook on teacher evaluation: assessing and improving performance. Larchmont, NY: Eye On Education.

Stufflebeam, D. L. (1983). The CIPP model for program evaluation. In Evaluation models (pp. 117-141). Dordrecht: Springer.

Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American journal of evaluation, 27(2), 237-246.

Tri-Partite Alliance. (2014). Work-based assessment: a practical guide. Building an assessment system around work. Retrieved from https://www.racp.edu.au/docs/default-source/default-document-library/worked-bases-assessment-racp.pdf?sfvrsn=b0242e1a_2

Tuytens, M., & Devos, G. (2017). The role of feedback from the school leader during teacher evaluation for teacher and school improvement. Teachers and Teaching, 23(1), 6-24.

Victoria State Government. (2019). Framework for Improving Student Outcomes (FISO). Retrieved from https://www.education.vic.gov.au/Documents/about/educationstate/fisoedstatefactsheet.pdf.

Wahlstrom, K. L., & Louis, K. S. (2008). How teachers experience principal leadership: The roles of professional community, trust, efficacy, and shared responsibility. Educational Administration Quarterly, 44(4), 458-495.

Weingarten, R. (2010). A New Path Forward: Four Approaches to Quality Teaching and Better Schools. American educator, 34(1), 36-39.

Weisberg, D., Sexton, S., Mulhern, J., Keeling, D., Schunck, J., Palcisco, A., & Morgan, K. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. Retrieved from

Wise, A. E., Darling-Hammond, L., McLaughlin, M. W., & Bernstein, H. T. (1985). Teacher evaluation: A study of effective practices. Elementary School Journal, 86(1), 61-121. doi:doi:10.1086/461437

Yarbrough, D. B., Shula, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The Program Evaluation Standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Corwin Press.

Zepeda, S. J. (2016). Principals' Perspectives: Professional Learning and Marginal Teachers on Formal Plans of Improvement. Research in Educational Administration & Leadership, 1(1), 25-59.


https://www.education.vic.gov.au/Documents/about/educationstate/fisoedstatefactsheet.pdf

https://www.racp.edu.au/docs/default-source/default-document-library/worked-bases-assessment-racp.pdf?sfvrsn=b0242e1a_2

https://www.racp.edu.au/docs/default-source/default-document-library/worked-bases-assessment-racp.pdf?sfvrsn=b0242e1a_2

9.AppendicesAppendix A: Literature Scan Criteria


Publication date Limit to published within the last 5 years

Sectors Human service professions, including teachers, social work and justice, allied health, clinical health, defence and army, emergency service personnel

Eligible studies Any of the following that include guidelines, criteria, frameworks or tools for professional practice evaluation.

Policy documents on existing evaluation tools

Theoretical and/or commentary articles on evaluating/assessing personnel evaluation tools

Existing reviews of personnel evaluation (narrative syntheses, systematic reviews and meta-analyses)

Psychometric studies on the quality and validity of personnel evaluation tools

Qualitative studies on the quality and validity of personnel evaluation tools

Countries Studies conducted in OECD countries

Language Studies in English language

Pre-conditions search terms

keywords and terms as tabled or documented as the following: “professional development” “training” “teacher” “school climate” “resources” “pre-requisites”

“supporting professional learning” “teacher evaluation” “limitations” “teacher evaluation”

“effective” “teacher agency” “self-assessment” “conceptions of assessment” “embed*” “system”

“structure” “support*” “implement*”

Theme 1 Theme 2 Theme 3 Theme 4 Theme 5

Professional

Growth

Learning

Development

Criteria

Barriers

Enablers

Facilitate

Conditions

Factors

Framework

Guidelines

Evaluation

Healthcare

Education

Allied Health

Clinical Health

System

Leadership

Attitudes


Appendix B: Organisations Represented at Workshops

Table 30. Organisations Represented at the Sydney Workshop (20th June 2019)

Organisation Name

Australasian Teacher Regulatory Authorities

Australian Catholic University

Australian Council of TESOL Associations

Australian Government Department Education and Training

Australian Professional Teacher Association

Independent Education Union of Australia

Primary English Teachers Association Australia

Queensland College of Teachers

Queensland Teachers Union

The University of Sydney

University of Technology Sydney

Table 31. Organisations Represented at the Darwin Workshop (27th June 2019)

Organisation Name

Australian Education Union (NT)

Casuarina Senior College

Northern Territory Government Department of Education

Teacher Registration Board of the Northern Territory

Table 32. Organisations Represented at the Melbourne Workshop (18th July 2019)

Organisation Name

ACT Teacher Quality Institute, ACT Government

Australian Institute of School Teaching and Leadership

Australian Mathematical Sciences Institute

Charles Darwin University

Education Services Australia

Educators SA

Federation University Australia


Organisation Name

Heatherwood School

History Teachers' Association of Victoria

Independent Education Union (Victoria/Tasmania branch)

Independent Schools Queensland

Mathematical Association of Victoria

Mercy College

Primary English Teachers Association Australia

St. Michael's Grammar School

Teachers Registration Board of South Australia

University of Tasmania, School of Education

Victoria University

Victorian Institute of Teaching

Table 33. Organisations Represented at the Perth Workshop (23rd July 2019)

Organisation Name

Canning Vale Primary School

Catholic Education WA

Curtin University

Department of Education, Western Australia

Independent Education Union of Australia

Penhros School

School of Special Educational Needs: Behaviour and Engagement

Sheridan College

Swan Valley Anglican Community School

University of Western Australia


Appendix C: Delphi Survey Data – Ranked Importance of Principles

The following section presents mean scores for each component within each of the eleven principles. Respondents were asked to rank the level of importance of each component on a 10-point Likert scale, where 1=least important, and 10= the highest level of importance.

Table 34. Mean Scores for Items in the Evidence-Based Principle (Round 1 and 2)

Item R1 Mean(SD)

R2 Mean (SD)

Carried out by educators who have comprehensive knowledge of the expected Standard(s)

9.05 (1.46) 8.38 (1.70)

Carried out by educators who have comprehensive knowledge of the skills to be assessed

8.98 (1.41) 8.65 (1.55)

Justified based on evidence contained in the assessment 8.91 (1.56) 8.62 (1.42)

Carried out by educators who have relevant expertise in the skills to be assessed

8.76 (1.50) 8.85 (1.08)

Carried out by educators who have comprehensive knowledge of performance assessment

8.74 (1.55) 8.19 (2.04)

Carried out by educators who have relevant expertise in the expected Standard(s)

8.69 (1.63) 8.69 (1.29)

Carried out by educators who have relevant expertise in performance assessment

8.48 (1.60) 8.15 (1.95)

R1 N=42; R2 N=26

Figure 22. Percentage of responses by item in the evidence-based principle (Round 1).


Figure 23. Percentage of responses by item in the evidence-based principle (Round 2).

Table 35. Mean Scores for Items in the Authentic Principle (Round 1 and 2)

Item R1 Mean(SD)

R2 Mean(SD)

Collected in a manner that ensures validity 9.21 (1.37) 8.65 (1.33)

Reflective of real-life practice 9.07 (1.06) 8.85 (1.29)

Collected in a manner that ensures transparency 9.07 (1.30) 8.69 (1.85)

Undertaken within the context of practice 8.91 (1.29) 8.19 (1.70)

Undertaken to capture evidence of knowledge 8.58 (1.42) 7.58 (1.72)

Undertaken to capture evidence of skills 8.49 (1.52) 7.58 (1.92)

Undertaken to capture evidence of attitudes 8.00 (1.62) 7.04 (2.20)

R1 N=43; R2 N=26

Figure 24. Percentage of responses by item in the authentic principle (Round 1).


Figure 25. Percentage of responses by item in the authentic principle (Round 2).

Table 36. Mean Scores for Items in the Relevant Principle (Round 1 and 2)

Item R1 Mean(SD)

R1 Mean(SD)

Should be relevant to the context 8.98 (0.99) 8.50 (1.50)

Should be linked to a diverse set of standards, competencies or performance metrics 8.51 (1.50) 7.54 (2.27)

R1 N=41; R2 N=26

Figure 26. Percentage of responses by item in the relevant principle (Round 1).


Figure 27. Percentage of responses by item in the relevant principle (Round 2).

Table 37. Mean Scores for Items in the Cost-Effective Principle (Round 1 and 2)

Item R1 Mean(SD)

R2 Mean(SD)

Consider training requirements for assessors 8.57 (1.48) N/A

Consider analytical expertise 7.88 (1.66) N/A

Consider administrative support 7.38 (1.96) N/A

Consider infrastructure requirements 7.31 (1.88) N/A

Consider costs of a specific assessment strategy or tool 7.26 (1.96) N/A

R1 N=42

Figure 28. Percentage of responses by item in the cost-effective principle (Round 1).


Table 38. Mean Score for Items in the Professional Growth Principle (Round 1 and 2)

Item R1 Mean(SD)

R2 Mean(SD)

Should provide actionable feedback to allow planning for career development 9.17 (1.14) 8.73 (1.54)

Should provide data that may be used to prioritise areas for professional growth 9.00 (1.14) 8.31 (1.38)

Should provide data that may determine areas for professional growth 8.98 (1.11) 8.27 (1.43)

Should provide data that may identify professional learning opportunities 8.66 (1.43) 8.04 (1.61)

R1 N=41; R2 N=26

Figure 29. Percentage of responses by item in the professional growth principle (Round 1).

Figure 30. Percentage of responses by item in the professional growth principle (Round 2).


Table 39. Mean Scores for Items in the Regular Principle (Round 1 and 2)

Item R1 Mean(SD)

R2 Mean(SD)

Be conducted to yield feedback on a variety of specific practices 8.24 (1.64) N/A

Require assessment of performance over time to assess competence 7.98 (1.80) N/A

Be conducted to provide ongoing progress monitoring 7.66 (1.87) N/A

Require regular accumulation of data or evidence to assess competence 7.44 (1.96) 7.35 (1.72)

R1 N=41; R2 N=26

Figure 31. Percentage of responses by item in the regular principle (Round 1).

Table 40. Mean Scores for Stems in the Context-Specific Principle (Round 1 and 2)

Item R1 Mean(SD)

R2 Mean(SD)

Be tailored to the practitioner's current practice context 8.77 (1.19) 8.27 (1.59)

Be tailored to practice settings 8.40 (1.50) 8.42 (1.53)

Be tailored to school context 8.21 (1.49) 8.23 (1.66)

R1 N=43; R2 N=26


Figure 32. Percentage of responses by item in the context-specific principle (Round 1).

Figure 33. Percentage of responses by item in the context-specific principle (Round 2).

Table 41. Mean Scores for Items in the Comprehensive Principle (Round 1 and 2)

Item R1 Mean(SD)

R2 Mean(SD)

Be based on a combination of methods to provide evidence 9.14 (1.05) 8.35 (1.72)

Involve the use of multiple samples of evidence 8.79 (1.22) 8.27 (1.73)

Involve the use of multiple modalities 8.14 (1.56) 8.35 (1.60)

Use direct evidence (e.g. witnessed by assessor) 8.14 (1.56) N/A

Use self-assessment 7.98 (1.76) N/A

Use indirect evidence (e.g. reviewed later) 7.74 (1.33) N/A

Use supplementary evidence (e.g. third-party reporting) 7.38 (1.82) N/A

R1 N=42; R2 N=26


Figure 34. Percentage of responses by item in the comprehensive principle (Round 1).

Figure 35. Percentage of responses by item in the comprehensive principle (Round 2).

Table 42. Mean Scores for Items in the Student-Based Principle (Round 1 and 2)

Item R1 Mean(SD)

R2 Mean(SD)

Entail observation of classroom interactions between the practitioner and the student(s)

8.49 (1.66) 8.15 (1.97)

Gauge and measure student engagement 7.95 (1.99) N/A

Gauge and measure student growth 8.27 (1.57) 7.42 (2.04)

Gauge and measure student understanding 8.46 (1.73) 7.54 (2.06)

Involve students in the feedback/evaluation process 7.56 (2.04) N/A

Involve student feedback on teacher practice 7.22 (2.32) N/A

Gauge and measure student interaction 7.10 (2.27) N/A

Gauge and measure student enjoyment 6.73 (2.30) N/A

Involve students in the measurement of teacher growth 6.46 (2.27) N/A


R1 N=41; R2 N=26

Figure 36. Percentage of responses by item in the student-based principle (Round 1).

Figure 37. Percentage of responses by item in the student-based principle (Round 2).

Table 43. Mean Scores for Items in the Collaborative Principle (Round 1 and 2)

Item R1 Mean(SD)

R2 Mean(SD)

Involve reflection on performance 9.37 (0.93) 8.65 (1.41)

Allow for collaboration between the assessor (e.g. Principal) and the practitioner to enable validation of judgements made

8.51 (1.58) 7.92 (1.77)

Involve high levels of communication between the assessor and individual being assessed

8.14 (1.98) N/A

Allow for collaboration between the assessor (e.g. Principal) and the practitioner 8.00 (1.73) N/A


Item R1 Mean(SD)

R2 Mean(SD)

Involve reinterpretation of performance by the practitioner 7.95 (1.73) 7.15 (1.51)

Allow for collaboration between peers 7.95 (1.65) N/A

R1 N=43; R2 N=26

Figure 38. Percentage of responses by item in the collaborative principle (Round 1).

Figure 39. Percentage of responses by item in the collaborative principle (Round 2).

Table 44. Mean Scores for Items in the Understandability Principle (Round 1 and 2)

Item R1 Mean(SD)

R2 Mean(SD)

Feedback must be clear and easy to understand for all parties 9.66 (0.62) 9.27 (1.28)

Purpose should be clear and easy to understand for all parties 9.59 (0.63) 8.77 (1.31)

Standards should be clear and easy to understand for all parties 9.44 (0.87) 8.77 (1.28)


Item R1 Mean(SD)

R2 Mean(SD)

Domains should be clear and easy to understand for all parties 9.39 (0.86) 8.65 (1.55)

Data should be clear and easy to understand for all parties 9.37 (0.89) 8.50 (1.66)

R1 N=41; R2 N=26

Figure 40. Percentage of responses by item in the understandability principle (Round 1).

Figure 41. Percentage of responses by item in the understandability principle (Round 2).

Table 45. Mean and Standard Deviation for All Items

Principle Item R1 Mean(SD)

R1N R2 Mean(SD)

R2N

Understandable Feedback must be clear and easy to understand for all parties

9.66 (0.62) 41 9.27 (1.28) 26



R1N R2 Mean(SD)

R2N

Understandable Purpose should be clear and easy to understand for all parties

9.59 (0.63) 41 8.77 (1.31) 26

Understandable Standards should be clear and easy to understand for all parties

9.44 (0.87) 41 8.77 (1.28) 26

Understandable Domains should be clear and easy to understand for all parties

9.39 (0.86) 41 8.50 (1.66) 26

Collaborative Involve reflection on performance 9.37 (0.93) 43 7.15 (1.51) 26

Understandable Data should be clear and easy to understand for all parties

9.37 (0.89) 41 8.65 (1.55) 26

Authentic Collected in a manner that ensures validity 9.21 (1.37) 43 8.65 (1.33) 26

Professional Growth Should provide actionable feedback to allow planning for career development

9.17 (1.14) 41 8.73 (1.54) 26

Comprehensive Be based on a combination of methods to provide evidence

9.14 (1.05) 42 8.35 (1.72) 26

Authentic Reflective of real-life practice 9.07 (1.06) 43 8.85 (1.29) 26

Authentic Collected in a manner that ensures transparency

9.07 (1.30) 43 8.69 (1.85) 26

Evidence-based Carried out by educators who have comprehensive knowledge of the expected Standard(s)

9.05 (1.46) 42 8.38 (1.70) 26

Professional Growth Should provide data that may be used to prioritise areas for professional growth

9.00 (1.14) 41 8.31 (1.38) 26

Evidence-based Carried out by educators who have comprehensive knowledge of the skills to be assessed

8.98 (1.41) 42 8.65 (1.55) 26

Relevant Should be relevant to the context 8.98 (0.99) 41 8.50 (1.50) 26

Professional Growth Should provide data that may determine areas for professional growth

8.98 (1.11) 41 8.27 (1.43) 26

Authentic Undertaken within the context of practice 8.91 (1.29) 43 8.19 (1.70) 26

Evidence-based Justified based on evidence contained in the assessment

8.91 (1.56) 42 8.62 (1.42) 26

Comprehensive Involve the use of multiple samples of evidence

8.79 (1.22) 42 8.27 (1.73) 26

Context Specific Be tailored to the practitioner's current practice context

8.77 (1.19) 43 8.27 (1.59) 26

Evidence-based Carried out by educators who have relevant expertise in the skills to be assessed

8.76 (1.50) 42 8.85 (1.08) 26

Evidence-based Carried out by educators who have comprehensive knowledge of performance assessment

8.74 (1.55) 42 8.19 (2.04) 26

Evidence-based Carried out by educators who have relevant 8.69 (1.63) 42 8.69 (1.29) 26



R1N R2 Mean(SD)

R2N

expertise in the expected Standard(s)

Professional Growth Should provide data that may identify professional learning opportunities

8.66 (1.43) 41 8.04 (1.61) 26

Authentic Undertaken to capture evidence of knowledge

8.58 (1.42) 43 7.58 (1.72) 26

Cost-effective Consider training requirements for assessors 8.57 (1.48) 42 N/A 26

Relevant Should be linked to a diverse set of standards, competencies or performance metrics

8.51 (1.50) 41 7.54 (2.27) 26

Collaborative Allow for collaboration between the assessor (e.g. Principal) and the practitioner to enable validation of judgements made

8.51 (1.58) 43 7.92 (1.77) 26

Authentic Undertaken to capture evidence of skills 8.49 (1.52) 43 7.58 (1.92) 26

Student-based Entail observation of classroom interactions between the practitioner and the student(s)

8.49 (1.66) 41 N/A 26

Evidence-based Carried out by educators who have relevant expertise in performance assessment

8.48 (1.60) 42 8.15 (1.95) 26

Student-based Gauge and measure student understanding 8.46 (1.73) 41 N/A 26

Context Specific Be tailored to practice settings 8.40 (1.50) 43 8.42 (1.53) 26

Student-based Gauge and measure student growth 8.27 (1.57) 41 7.42 (2.04) 26

Regular Be conducted to yield feedback on a variety of specific practices

8.24 (1.64) 41 N/A 26

Context Specific Be tailored to school context 8.21 (1.49) 43 8.23 (1.66) 26

Comprehensive Involve the use of multiple modalities 8.14 (1.56) 42 8.35 (1.60) 26

Comprehensive Use direct evidence (e.g. witnessed by assessor)

8.14 (1.56) 42 N/A 26

Collaborative Involve high levels of communication between the assessor and individual being assessed

8.14 (1.98) 43 N/A 26

Collaborative Allow for collaboration between the assessor (e.g. Principal) and the practitioner

8.00 (1.73) 43 N/A 26

Authentic Undertaken to capture evidence of attitudes 8.00 (1.65) 43 7.04 (2.20) 26

Comprehensive Use self-assessment 7.98 (1.76) 42 N/A 26

Regular Require assessment of performance over time to assess competence

7.98 (1.80) 41 N/A 26

Collaborative Involve reinterpretation of performance by the practitioner

7.95 (1.73) 43 7.92 (1.77) 26

Collaborative Allow for collaboration between peers 7.95 (1.65) 43 N/A N/A

Student-based Gauge and measure student engagement 7.95 (1.99) 41 N/A N/A

Cost-effective Consider analytical expertise 7.88 (1.66) 42 N/A N/A



R1N R2 Mean(SD)

R2N

Comprehensive Use indirect evidence (e.g. reviewed later) 7.74 (1.33) 42 N/A N/A

Regular Be conducted to provide ongoing progress monitoring

7.66 (1.87) 41 N/A N/A

Student-based Involve students in the feedback/evaluation process

7.56 (2.04) 41 N/A N/A

Regular Require regular accumulation of data or evidence to assess competence

7.44 (1.96) 41 7.35 (1.72) 26

Cost-effective Consider administrative support 7.38 (1.96) 42 N/A N/A

Comprehensive Use supplementary evidence (e.g. third-party reporting)

7.38 (1.82) 42 N/A N/A

Cost-effective Consider infrastructure requirements 7.31 (1.88) 42 N/A N/A

Cost-effective Consider costs of a specific assessment strategy or tool

7.26 (1.96) 42 N/A N/A

Student-based Involve student feedback on teacher practice

7.22 (2.32) 41 N/A N/A

Student-based Gauge and measure student interaction 7.10 (2.27) 41 N/A N/A

Student-based Gauge and measure student enjoyment 6.73 (2.30) 41 N/A N/A

Student-based Involve students in the measurement of teacher growth

6.46 (2.27) 41 N/A N/A

Professional Agency Be owned by the teacher as their personal professional development journey

N/A N/A 8.85 (1.57) 26

Professional Agency Be selected based on teachers' individual professional judgment of their growth needs

N/A N/A 7.77 (1.90) 26

Professional Agency Be directed by the teacher based on self-identified areas of development

N/A N/A 7.70 (2.01) 26

Professional Agency Be informed by collaborative feedback with other colleagues

N/A N/A 7.35 (1.79) 26

Professional Agency Be informed by participative feedback with other colleagues

N/A N/A 7.19 (2.17) 26

Professional Agency Not be solely driven by other school improvement agendas

N/A N/A 7.15 (2.69) 26

Professional Agency Be directed by the teacher based on self-identified strengths

N/A N/A 7.13 (2.16) 26

Professional Agency Not be compliance-based N/A N/A 7.00 (2.62) 26

Professional Agency Be directed by the teacher based on interests

N/A N/A 6.46 (2.28) 26

Professional Agency Be guided by teachers' personal criteria for assessment

N/A N/A 6.42 (2.34) 26

Utility Outputs (e.g. data, feedback) should be used by the teacher for informing their own

N/A N/A 7.92 (2.00) 26



R1N R2 Mean(SD)

R2N

professional growth journey

Utility Should be selected by the teacher to identify growth opportunities for career development

N/A N/A 7.42 (2.00) 26

Utility The teacher should determine whether assessment data is shared with leadership personnel to identify opportunities for career development

N/A N/A 6.39 (2.58) 26

Utility Should be determined by the practitioner as to whether this occurs in collaborative partnership with leadership personnel

N/A N/A 6.35 (2.43) 26

Utility The teacher should decide whether assessment data is used by leadership personnel to identify opportunities for career development

N/A N/A 5.69 (2.78) 26

Utility The teacher should decide whether assessment data is shared with leadership personnel to identify alignment with school improvement objectives

N/A N/A 5.58 (2.75) 26

Utility The teacher should decide whether assessment data is used by leadership personnel to identify alignment with school improvement objectives

N/A N/A 5.12 (2.70) 26


Appendix D: Teacher Evaluation Tools in Australian Contexts

Table 46. Evaluation Tools and Methods (in collaboration with AITSL) from the Investigating the Key Characteristics of Effective Teachers Review

Evaluative Method (i.e. Data Collection Method)

Evaluation Tools and Strategies

Teachers (i.e. teacher self-reflection, self-efficacy, teacher self-rating, teacher interviews)

Teacher Self-Assessment Tool (TSAT)

Visible Learning Matrix

Teaching for Effective Learning compass (TfEL)

Educator Impact – 360 Degree Reflect Tool

Classroom Strategies Scales-Teacher form (CSS-T)

Teacher Evaluation Questionnaire (TEQ)

Certification procedures i.e. professional discussion, pre-observation

Registration procedures i.e. portfolios

Observation (i.e. classroom observation/teaching practice)


Swivl Video Observations

Teacher Tracker tool (T3) / Collaborative Approach to Teaching Observations (CATO)

Danielson Framework for Teaching Evaluation Instrument (FfT)

Marzano Teacher Evaluation Model

Measures of Effective Teaching Project

Classroom Assessment Scoring System (CLASS)

iWalk Observation

eObservations (online classroom observation tool)

Classroom Observation Tool Suite

OBSeRVE (Teacher Observation Software Tool)

Simulated scenario: Objective Structured Teaching Exercise (OSTE)

Classroom Strategies Scales-Teacher form (CSS-T)


Peers (i.e. peer reflection, peer collaboration, appraisals, mentoring, peer rating)


Educator Impact – 360 Degree Reflect tool



Registration procedures

Leader (i.e. evaluation by third party,




Evaluative Method (i.e. Data Collection Method)

Evaluation Tools and Strategies

principal/supervisor/school leader)Educator Impact – 360 Degree Reflect tool

Students(i.e. student perception and student ratings)

Pivot Professional Learning

Tripod (student feedback for K-12 educators)

Student perception of teaching questionnaire (SPTQ)

Artefacts (i.e. evidence of impact or classroom impacts)

Teacher Performance assessment (edTPA)

Intellectual Demand Assignment Protocol (IDAP)

Instructional Quality Assessment (IQA)

Quality Assessment in Science (QAS) Notebook

Scoop Notebook and Rating Guide


Appendix E: Prompts for Principles

This document is intended to be a practical resource for teachers, school leaders and the system when engaging in a process of teacher evaluation. It presents an overview of the principles to inform tool and resource selection and evaluation practice, and an accompanying list of relevant prompts or questions to ask when engaging in this process. The questions and prompts are intended to be used or referred to throughout the evaluation process, for example for when determining which of the rubric standards are necessary to be met, in order to inform tool selection. Though not intended to be an exhaustive list, it assists users in identifying the key priority areas of focus for users when selecting and planning for evaluations to promote teachers’ professional growth.

The principles are presented in three groups: content, purpose, and process, each of which are associated with relevant prompts or questions to ask. Altogether, the combination of these groups is aimed at enhancing teacher practice and student outcomes, both of which can be considered the product of the evaluation.

Cont

ent

Principles in this grouping (relevant, understandable, authentic, context-specific) describe the characteristics of content that should be captured in an evaluation, assessment or feedback resource that could be used effectively in a system that supports professional growth.

Principles Who? Prompts

Rele

vant

, Und

erst

anda

ble,

aut

henti

c, c

onte

xt-s

peci

fic

Authentic: Does the tool capture what I intend to measure? Does the tool reflect real-life practice? Relevant: Will the tool provide relevant information regarding my practice and my professional growth goal, as aligned with the

APST? Understandable: Is there a shared understanding between myself and my colleagues (i.e. peers, faculty leaders, school leaders)

around the a) scope of evaluation, b) purpose of evaluation, and c) future use of data, feedback and outputs? Context-specific: Can this tool be tailored for use in my classroom or practice context?

Authentic: Is the use of the tool being used in a way that ensures honesty and validity of judgements on real-life teacher practice? Relevant: Will this tool provide the necessary, relevant information for this teacher and in alignment with the broader school

improvement agenda? Understandable: Have expectations around the purpose of evaluation, and any benchmarks been clearly communicated and

understood? Context-specific: Is this tool appropriate for use in this school setting?

Authentic: Do the evaluation tools used in Australian schools measure practice (skills, attitudes, and knowledge) that is reflective of real-life practice?

Relevant: Are current teacher evaluation practices (either formal or informal) relevant to the current practice contexts of Australian teachers?

Understandable: Is there consistent and clear messaging on the purpose of teacher evaluation and intended outcomes, that is focused on ensuring professional growth?

Context-specific: How applicable is the current range of teacher evaluation tools to the different jurisdictional, regional, and school


T

SL

SYS

contexts in Australia?Pu

rpos

e

Principles (collaborative, evidence-based, professional growth) that describe the intentional characteristics of evaluation, assessment or feedback that would support teachers’ professional growth.

Principles Who? Prompts

Colla

bora

tive,

evi

denc

e-ba

sed,

pro

fess

iona

l gro

wth

Collaborative: At which stage during the evaluation should a collaborative conversation take place with my colleagues (e.g. during pre-assessment, during the assessment, or in the interpretation or reflection on performance)?

Evidence-based: Is this tool standardised, evidence-based, and reliable? Is the tool used by an assessor who is skilled or has been trained in assessment?

Professional growth: Will the tool (through its use or the provided outputs) allow me to determine and prioritise areas for professional growth? Does the tool provide actionable feedback?

Collaborative: What level of input, from a leadership perspective, is necessary throughout the stages of the evaluation (e.g. pre-assessment, during the assessment, or in the interpretation or reflection on performance)?

Evidence-based: Are there trained assessors at the school who can use the tool, or are external supports necessary?

Professional growth: What supports can the school provide the teacher before, during and after the evaluation to ensure that this process can facilitate professional growth?

Collaborative: To what extent should evaluative practices be embedded within, or mirror collaborative structures at schools (e.g. Professional Learning Teams, professional learning communities, communities of practice?)

Evidence-based: How are evaluations currently conducted in Australian schools? What can be said about the assessors carrying out informal and informal assessments in schools? What can be said about the state or number of trained assessors (at a school level) who are making judgements of teacher practice against the APST?

Professional growth: Is the language, purpose/focus, and type of teacher evaluation tools and resources sufficiently focused on promoting teachers’ professional growth?

Process


T

SL

SYS

Principles (regular, student-based and comprehensive, professional agency, utility) that describe the characteristics of conducting evaluation, assessment or feedback in a way that would support teachers’ professional growth.

Principles Who? PromptsRe

gula

r, st

uden

t-ba

sed

and

com

preh

ensi

ve, p

rofe

ssio

nal

agen

cy, u

tility

Regular: Does the tool provide data or feedback on my progress over multiple periods of time, to provide a snapshot of my practice and professional growth journey?

Student-based: Is it important for my students to be involved in the feedback/evaluation process, and provide insight into an area of my practice?

Comprehensive: Does the tool collect multiple samples of evidence or require multiple perspectives to make a judgement of my practice (either formative or summative)?

Professional agency: Is the tool, and its data, feedback and outputs provided informed by my personal criteria for assessment?

Utility: Is there a clear link between the data/information provided by the tool, and my identified area of professional growth and learning?

Regular: What data or evidence is necessary to be collected to add to, supplement or contribute to what is already known or existing about the teacher’s professional growth journey?

Student-based: Is it important for teacher evaluation at this school to incorporate student voice?

Comprehensive: How many modalities, samples of evidence, or assessors are required to make an evidenced judgement of the identified goal? What resources are available for evaluation?

Professional agency: What type of supports or professional judgement can I provide to my teacher throughout this process?

Utility: To what extent should school leadership have involvement, or contribute towards the teacher’s use of the tool, and any feedback or data that is provided?

Regular: What are the jurisdictional requirements for conducting teacher evaluation and assessment?

Student-based: To what extent is student-based evaluation tools available and used across schools, regions, and states and territories?

Comprehensive: What are the requirements for evaluation for different career stages and purposes (e.g. Highly Accomplished and Lead Teacher certification, obtaining full registration)?

Professional agency: Are there appropriate guidelines to facilitate use of teacher evaluation tools and resources in a way that promotes teacher professional agency?

Utility: To what extent do the teacher evaluation tools and resources used in Australian schools provide actionable feedback for supporting teachers’ professional growth?


T

SL

SYS

Professor Janet ClintonE: [email protected]

P: +61 3 9035 3697
