the program review process: ncate and the state of indiana richard frisbie and t. j. oakes march 8,...

The Program Review Process: NCATE and the State of Indiana

Richard Frisbie and T. J. OakesMarch 8, 2007(source:NCATE, February 2007)

Programs for which NCATE does not have standards

Art, music, dance, or drama education Business, speech, and vocational

education Some advanced teacher education

programs (e.g., Curriculum & Instruction School counselorThese programs undergo state review or

are accredited by other accrediting agencies

NCATE Program Review-Specialized Professional Associations (SPAs)

The national organizations that represent teachers, professional education faculty, and other school personnel who teach a specific subject matter, teach students at a specific developmental level, teach students with special needs, administer schools, or provide services to students.

State of Indiana Program Review

The state review is similar to the NCATE process. For details on the state review, see http://www.doe.state.in.us/dps/teacherprep/program_review.html

Challenges for Institutions

Programs are having surprising difficulty demonstrating how assessments align with SPA standards

Programs have had difficulty selecting appropriate assessments

6-8 Assessments: The Rules

Institution must submit a minimum of six assessments, unless the SPA specifies more than six required assessments

Institution may submit additional assessments when SPA does not specify all eight assessments

Five specific types of assessments are required by all SPAs

Required Assessments in National Program Review

1. State licensure exam for content2. Assessment of content knowledge3. Assessment of Planning (e.g., lesson

plan or unit plan)4. Student teaching/internship assessment5. Assessment of candidate impact on

student learning or providing a supporting learning environment

6. Other assessment

NCATE Unit Standard #1 Candidates preparing to work in

schools as teachers or other school personnel know the content of their fields, demonstrate professional and pedagogical knowledge, skills, and dispositions and apply them so that students learn. Assessments indicate that candidates meet professional, state, and institutional standards.

Required Program Assessments

Content:1. State licensure exam for program

area (if available—otherwise another content based assessment)

2. Content AssessmentProfessional and Pedagogical

Knowledge Skills and Dispositions:3. Assessment of Planning (e.g., unit

plan)

4. Student teaching/internship assessment

P – 12 Student Learning:5. Assessment of candidate impact on

student learning or providing a supporting learning environment

Program is required to have at least one more assessment, can have a maximum of 8 to demonstrate candidate mastery of SPA standards

Some SPAs have additional required types

Important things learnedfrom the pilots:

Unit-wide assessments can be used as key program assessments, but faculty must demonstrate alignment with specific SPA standards

Three effective strategies

SPA-specific addendum SPA-specific rubrics Coding assessment

AND including information in narrative about context

Important things learned:

It is critical to document alignment between assessments, scoring guides and SPA standards

Most Important Learning from Pilots: All Parts of Assessments Must Interrelate!

Scoring Guide

Data & Findings

Assessment

STANDARDS

What does that mean?

Parts of the assessment are aligned with the SPA standards

These standards can be seen in the elements of the scoring guide

The data is broken down by the elements in the scoring guide

Developing Appropriate Assessments: What are the Criteria?

Working definitions of assessment

The term “assessment” can refer to different aspects of the program review process

Any of the 5 required types of assessment

A specific element of an assessment Assessment (assignment summary) Scoring guide Data & findings

The five key principles by which reviewers evaluate your reports

Are assessments aligned with standards? Do assessments meet the cognitive

demands and skill levels of the standards? Are assessments free from bias? Do scoring guides clearly describe the

different categories? Do data provide evidence for meeting

standards?

Distributing assessments across standards

Do not take a “this assessment addresses all standards” approach

Think of assessments as “primary” vs. “supportive” in meeting standards

Typically, one to three assessments, taken as a whole, meet a single standard

The assessment instrument

Align to standards! Preferably with references to standards in the assessment itself.

If alignment is not embedded in the assessment, provide alignment chart in your discussion of the assessment, and a strong argument for how the assessment covers the standard. Be specific.

Characteristics of a strong assessment

The objectives candidates must meet correspond to the depth and breadth required by SPA standards

Assessments are scored primarily on standards-based criteria.

The required activities candidates must perform are clearly described.

Assessment shows no evidence of bias.

Scoring Guides

must be consistent with the assessment instrument

acceptable scores must target the level of competence required by a standard

must clearly describe differences among scored categories

Putting it all together (with data)

The data table is how you “tell your story”

Data should be collected and used in ways that help your program analyze its strengths and weaknesses.

Data should mirror the scoring categories for the assessment (e.g. if grades are used, data should be reported as grades).

Speaking of grades…

Grades in required courses are usually acceptable for Assessment #2.

Provide course descriptions and align courses to standards.

Course content must intensively and comprehensively address standards.

Show range of grades received by candidates, not just GPAs.

Other comments on data

Remove identifying #s and names.Make sure the number of candidates

in data sets are consistent across tables in reports, or explain inconsistencies that might puzzle reviewers.

Note and explain any conversions of data (e.g. numerical scores converted to grades in a data table).

Analyze your data

Make the most of the two pages you have per assessment to analyze and “defend” your assessments and data (Section IV). Analyze your data according to the SPA standards. Compare data across standards, cohort groups, time periods, etc. Tell what you learn from it.

In Section V, make sure to describe what you are doing or will do to address areas where data are poor or below average.

How Much Data?

Fall 2008 and spring 2009, two

years of data Three years of data are optimal –

not necessary to provide more Remember: Data are only as good

as the assessment/scoring guide

Resources on NCATE web site www.ncate.org, click on Institutions,

click on Program Reviews Program Report Forms, Guidelines,

Instructions, Examples of Assessments, Examples of Report Sections, Examples of Reports

www.ncate.org, click on Program Reviewers All training materials for program

reviewers

Who to Contact at SPAs

For questions related to the standards and what are appropriate assessments for a particular discipline:

SPA Program Review Coordinators http://www.ncate.org/programreview/spacontact.asp?ch=88