3-toolsinstrumentspreval

8/14/2019 3-TOOLSINSTRUMENTSPREVAL

1/36

Tools and Instruments of Program Evaluation 1

Running head: TOOLS AND INSTRUMENTS OF PROGRAM EVALUATION

Tools and Instruments of Program Evaluation: An Analysis

Richard Joffray

Argosy University

Advanced Supervision of Curriculum and Instruction

L. Warner

April 9, 2008


2/36


Abstract

Various tools and instruments are needed in a successful educational program evaluation. The

Seattle Art Institute (AIS) believes in and commit to principles of learner-centered (L-C)

instruction. The simple definition of L-C instruction has been that it focuses on what the

student learns and not what the instructor teaches. Evaluation in the information-centered

paradigm is most often used for assessment oflearning; in the learning-centered paradigm,

assessment isforlearning. In other words, assessment in the L-C paradigm is almost always

formative, not summative. To this end, various techniques of data collections, opinion surveys

and direct observations are used to evaluate and adjust the learning environment presented to

the students. Curriculum and class lessons are based on core competencies needed by

students in a real-world environment. Professional educators together with industry specialists

and potential employers decide the future of these needs and attempt to prepare their students

for the challenges of an ever-changing work place.


3/36


Table of Contents

Introduction ....................................................................................................................... 4

Sequence of Events ........................................................................................................... 4

Monthly ................................................................................................................. 4

Bi-Quarterly .......................................................................................................... 5

Quarterly ............................................................................................................... 5

Bi-Annually ........................................................................................................... 5

Annually ................................................................................................................ 5

Aspects and Procedures Of Evaluation ............................................................................. 6

Curriculum ............................................................................................................ 6

Instructional content .............................................................................................. 8

Teaching methods and strategies ........................................................................... 8

Evaluation process ................................................................................................ 9

School community ................................................................................................ 9

Tools and Instruments....................................................................................................... 10

Self Evaluation...................................................................................................... 10

Classroom Observation ........................................................................................ 10

Teaching Portfolio ................................................................................................ 11

Student Course Evaluation ....................................................................................11

Development Plan ................................................................................................ 12


4/36


5/36


Tools and Instruments of Program Evaluation: An Analysis

The Seattle Art Institute (AIS) believes in and commit to principles of learning-centered

(L-C) instruction. The simple definition of L-C instruction has been that it focuses on what the

student learns and not what the instructor teaches. In a learning-centered classroom, everyone

learns and everyone teaches. It is also a student outcomes-oriented organization. They strive for

the maximum graduates opportunities for entry into various professions as well as advancement

in the fields for which they prepared. Because the curricula and faculty focus on the

development of industry-specific exit competencies, students can achieve high levels of skills

appropriate to their fields.

The process of achieving this graduating student involves preparation, delivery and

evaluation of curriculum in such a way that students, faculty and administrators can continually

evolve in a learning environment that prepares students for a professional working world.

Sequence of Events

Monthly. Program director meetings take place either digitally or in person to discuss the

curriculum content between schools

Bi-Quarterly. Department faculty meets to discuss the interactions of respective classes

and how they might overlap. Evaluations are written down and department heads take them to

their national meetings.

Quarterly. IDEA Questionnaires, which are designed with curriculum outcome

statements to provide direct feedback from students, are given. Department faculty meet to


6/36


discuss curriculum overlapping, teaching styles and general collaboration. A survey is given to

students, exiting graduates and alumni to determine any shortcomings that they might have about

the curriculum, school operations or general feedback.

Potential employers are also surveyed to get information about the state of their industry and exit

competencies that may be required of graduates.

Bi-Annually. A Professional Appraisal Committee(PAC) meeting is setup to find out

from the professionals in their industry, things that may be included in the catalog, industry

standards expected of graduating students and general feedback regarding interviews that they

might have had with graduating students.

Annually. Performance Review (PPAR), Self Evaluation Form, Department Head

Evaluation Form, Department Head Recommendations, and IDEA outcomes are all a yearly

ongoing evaluation data source to be reviewed as part of the requirements for a faculty letter of

appointment.

Aspects and Procedures Of Evaluation

Curriculum. Each time a new program is being considered for The Art Institutes or an

existing program is going to be evaluated and revised, a Curriculum Task Force may be

convened. A Curriculum Task Force is generally composed of one Dean of Academic Affairs

(usually the Chairperson), faculty members in the specific program, and Academic Affairs Staff.

Representatives from Admissions, Instructional Technology, and Career Services may participate

in the Curriculum Task Force as well.


7/36


Once a Curriculum Task Force is convened, the Chair generally conducts conference calls to

discuss issues related to specific curriculum development and revision issues. The Curriculum

Task Force may complete preliminary work related to program mission, program need, potential

enrollment analysis, employment outcomes, and salary levels prior to the meeting. These

meetings generally last five days. During the meetings, program exit competencies are developed

or revised (depending on the nature of the development or revision), course competencies are

established, portfolio requirements and evaluation criteria are determined, and course

descriptions are developed. The Ai Curriculum Development Guide, which contains information

about The Art Institutes Curriculum Model and a step-by-step description of the curriculum


8/36


development process, is used by Curriculum Task Force Members in their work. The curriculum

development work is usually finished after the meeting when faculty return to their schools

(Figure 1) Conference calls generally are scheduled to provide updates until all work is

completed. The Art Institute faculties are encouraged to take part in the curriculum development

and revision processes (EDMC, 2008).

Figure 1. Curriculum Development Flowchart

Curriculum Development Flowchart

(During Task Force Meeting)

Develop Program Mission Statement

Develop Program Exit Competencies

Label Program Exit Competencies: Foundation/Work-Ready

Develop Course (Enabling) Competencies


9/36


Bundle Course (Enabling) Competencies

Label Bundles: Core/Related Studies/General Education

Name Courses and List Course Competencies

Develop Portfolio Standards

Develop Course Descriptions

Develop Course Outlines

Develop Syllabi

Develop Lesson Plans

Instructional content. The faculty themselves are largely responsible for all of the

instructional content presented in the classroom. Their department heads, peer faculty and the

professional community in the fields that they teach in developing this content aid them. The


10/36


department heads in each discipline basically does evaluation and recommendations. Procedures

used in this process include, classroom observation, faculty interviews and reviews of individual

faculty development plans.

Teaching methods and strategies. Through an ongoing faculty development process at

AIS, instructor methods and strategies of teaching are developed. Quarterly an all faculty

presentation is given on industry standards, techniques and philosophies used in current teaching.

Through individual breakout sessions, faculty members meet to discuss commonalities in

teaching styles and future methods that can be employed by faculty members.

Evaluation process. Faculty appraisal worksheets, classroom observation, a teaching

portfolio, student course evaluations and a faculty development plan are the five main methods

used in a faculty evaluation process. Curriculum and faculty evaluation culminates the

supervisory process by evaluating instructional techniques, student opinion and instructor

qualifications by evaluating opinions and data, which ultimately issues a letter of appointment to

each faculty member for a term of service.

School community. A schools culture is one of the most important elements in its success

or failure in educating its students. The culture and climate of a school can be affected by factors

from disciplinary problems and classroom rowdiness to educator pessimism or student apathy.

Culture and climate, however, can most nearly be described as the sum of all perceptions and

emotions attached to the school, both good and bad, held by students, faculty, administrators,

parents, and the community at large. Every school is perceived as different and every school has


11/36


its own atmosphere and mood. Each year the human resources department to gather basic school

climate data conducts a faculty candy-bar survey along with student surveys. This data is

tabulated against the national set of schools and each school is scored accordingly to get a picture

of the schools general culture and climate.

Tools and Instruments

The Faculty Performance Planning and Appraisal Review (PPAR) synthesizes

information from the five sources of input summarized below and documents each faculty

members performance. The PPAR form reflects faculty key job elements and performance

expectations (EDMC, 2008).

Self Evaluation. This can be one of the most important parts of the formative evaluation

process for teachers functioning at moderate or high levels of development, expertise, and

commitment (Glickman et al, 2007). Each faculty member is requested to submit a self-

evaluation prior to the annual performance review date. This self-evaluation provides an

opportunity to summarize activities undertaken during the past year, reflect on performance

difficulties, and establish development goals. This information, when used in conjunction with

the other performance input sources, provides an annual update on the faculty members

accomplishments and identifies future development needs (EDMC Manual, 2008).

Classroom Observation. A formative observation instrument to describe what is

occurring in a classroom is a means for professional growth and instructional improvement

(Glickman et al, 2007). Formative observation is usually conditioned on prior agreement as to


12/36


what is needed to be learned in the classroom while a summative observation is based on

externally imposed criteria used to measure teacher improvements against one another. Faculty

members at AIS are observed in the classroom for both summative and formative assessments of

classroom performance. Faculty members are generally observed at least once a year by the

faculty members instructional supervisor. The class observer generally schedules the observation

with input from the faculty members. The observer generally meets with the faculty member to

provide feedback from the class observation (EDMC, 2008). The instrument used is

Teaching Portfolio. Each faculty member creates and maintains a teaching portfolio.

Mandatory items include a curriculum vitae, a statement of teaching philosophy, sample syllabi,

and examples of student work for each class taught. Optional items include examples of

professional work, published articles, or other items that provide insight into an instructors

teaching capabilities (EDMC, 2008).

Student Course Evaluation. Student empowerment through course evaluation can be an

instrument that reduces student frustration, allows students to feel a part of the school and an

enhancement to the overall school climate. AIS uses the Student Reaction to Instruction and

Course system developed by the IDEA Center at Kansas State University. This instrument

focuses on student learning and provides comparative data from colleges and universities

throughout the country. Generally, each quarter, each full-time faculty is evaluated in at least one

course. The courses faculty members are evaluated in generally follow a rotation throughout the

year. Part-time faculty members generally are evaluated at least once a year (EDMC, 2008).


13/36


Development Plan. In this age of rapid changes in information, technology as well as

teaching strategies and methodologies, teachers are in constant need of additional information

and techniques. Faculty members work with their instructional supervisors to generate their

individual development plan for the coming year. The plan provides a tool to assess, evaluate, and

plan learning and growth for faculty.

This plan builds connections between faculty experiential credentials and academic

credentials, establishes a plan for assessing performance, and identifies areas for developmental

opportunities. In addition, this plan is also designed to provide a vehicle for faculty to establish a

path for acquiring additional credentials if appropriate, match individual goals with institutional

ones, and build a documented portfolio to showcase ongoing teaching and professional growth.

Included is the opportunity to document faculty development in-field and out-of-field, as well as

list courses taken that are designed to facilitate the collegiate role of teacher.


14/36


Figure 2. Faculty Development Process

The developmental plan is designed to support the philosophy that learning is a lifelong process

for everyone and should be measured in a consistent, ongoing manner. The faculty members and

their instructional supervisor create individual faculty plans (EDMC, 2008).

Conclusion

For this analysis of the evaluation system to be complete we must look at the goals and

beliefs of the school as a whole and compare it to the evaluation system that is in place to

determine its success or failure.

Instruction. The goals for instruction are teachers prepare students for a professional


15/36


environment through a predefined set of exit competencies in a learner-centered instructional

environment. The use of industry standards as an instructional guideline prepares students for

placement upon graduation and fulfills the stated school goals. The strong school belief system

toward learner-centered education is reflected in the instructional assessment system.

Assessment. Using outcomes-oriented formative assessments of faculty teaching and

student work, this school is addressing well-researched strategies through the use of teacher self-

evaluation and classroom observation. This coupled with the teacher presentation of a teaching

portfolio is well suited addressing reflective strategies in teacher-based assessments. Both the

school and the teacher are teaching desired student outcomes by example.

Staff development. Ongoing faculty development is on the forefront of concerns at this

school and adheres to an Association of Intervention Specialist Certification Board (AISCB)

rules standard to form a solid Faculty Performance Evaluation System. Through the use of self-

evaluation and exploration, the system in place addresses concerns that teacher empowerment

(Argosy,2008), industry standards as well as modern ideologies of educational staff development.


16/36


References

Glickman, C.D., Gordon, S.P., & Ross-Gordon, J.M. (2007). Supervision and instructional

leadership:A developmental approach (7th ed.). Boston, MA: Pearson.

The Art Institute of Seattle & Education Management Corporation, (2008).Art Institutes

Academic Manual, Pittsburgh, PA

Argosy University (2008). E6350:Advanced Supervision of Curriculum and Instruction:

Module 1-7: Resource Pack. Retrieved April 13, 2008, from

http://myeclassonline.com/
http://myeclassonline.com/http://myeclassonline.com/


17/36


Appendix

JOB DESCRIPTIONS:Academic Affairs

Title: FACULTY, Full time (EDXXX50.FT)

Department: Academic Affairs

Position Summary:

Instruct and facilitate meaningful learning of the course competencies in the curriculum and

proactively support all facets of the learning environment. Provide education through learning-

centered instruction that will enable graduates to fulfill the evolving needs of the marketplace.

Encourage a culture of learning that values mutual responsibility and respect, life-long learning

and ethics as well as personal and professional development.

Key Job Elements:

Provides competency-based education

Designs/delivers class instruction

Enables student exit competencies

Delivers learning-centered instruction

Promotes student success


18/36


Manages the learning environment

Contributes to a culture of learning

Relates professional/life/industry experience to learning

Organizational Relationships:

Reports to: Dean of the School of, Department Chair/Academic Department Director

VP/Dean of Academic Affairs

Collaborates with: Academic Affairs Department

Other school/campus functional areas

Curriculum Task Forces and other school committees

Competencies and Expectations:

Key Job Element ACCOUNTABILITY PERFORMANCE EXPECTATION

Provides

Competency-

Based Education

Graduate Outcomes

Persistence Rate

Employment Rate

Starting Salaries

Provides instruction which creates the

opportunity for the student to achieve

outcomes

Aligns with the EDMC models for

curricula


19/36


Supports the EDMC style of system

delivery

Designs/Delivers

Class instruction

Syllabus/Lesson Plans

Classroom Assessment

Develops instructional plans to meet course

competencies

Develops activities which support lesson

objectives

Delivers online instruction as approved

(online instructors only)

Enables Exit

Competencies

Student Achievement

Evaluation of Learning

Provides instruction which fosters student

achievement of identified exit competencies

Establishes student performance criteria

and evaluation based on exit competencies

Delivers

Learning-

Centered

Instruction

Active Learning

Student-Oriented

Establishes a classroom environment

conducive to collaborative learning and

active student involvement

Prepares for classes to ensure student


20/36


success

Demonstrates effective planning and

preparation

Exhibits flexibility in style and work

schedule

Promotes Student

Success

Persistence

Student Satisfaction

Helps students solve problems that may

impede successful program completion

through advising, tutoring, mentoring,

and/or referring students on academic or

personal matters

Participates in student persistence activities

Exhibits passion for teaching and students

Engages students in the learning process

Meets students as individuals rather than a

group

Thinks outside traditional academic

processes to bring information to students

Champions student concerns as they relate


21/36


to student success

Participates in activities that build the

reputation of the school

Manages the

Learning

Environment

External/internal

Audits

(including

accreditation, Title IV

and other regulatory

compliance visits)

Keeps accurate records

Submits grade and other reports on time

Enforces school/campus academic and

attendance policies

Demonstrates management skills in the

classroom

Demonstrates lesson management in the

classroom

Maintains classroom in which all students

are treated in an equitable manner

Contributes to a

Culture of

Learning

Personal Development

Service

Participates on curriculum task forces as

needed

Participates on system task forces as needed


22/36


Supports local school/campus events such

as Open House, orientation, graduation, etc.

Participates in departmental meetings,

workshops, committee sessions, activities

and initiatives

Relates

Professional/Life/

Industry

Experience to

Learning

Experiential

Knowledge

Continues to develop professional/technical

skills

Introduces industry perspective into course

materials as appropriate

Maintains active awareness of

professional/industry trends and

opportunities

Position Requirements/Preferences

Appropriate Credential(s) Has sufficient credits in major area

Meets accreditation criteria for major area


23/36


(note: Faculty are required to seek higher credentials

when so instructed; ex. school changes accreditation

status)

Organizational Skills Handles and meets multiple deadlines

Develops projects without continuing direct

supervision

Reports necessary data and information timely and

accurately

Professional Criteria Demonstrates excellent written and oral

communications skills

Communicates in Standard English with correct

usage and a professional vocabulary

Is able to develop a professional rapport with diverse

school/campus constituents

Demonstrates good conflict resolution skills

Learning-Centered Demonstrates student-focused style

Exhibits willingness to learn from students


24/36


25/36


school and community

Possesses a high energy level

Is able to see beyond his/her courses to the vision of

the school as a whole

Considers him/herself a part of the Academic Affairs

team rather than a single faculty member

Personal Characteristics Enjoys learning

Exhibits professional behavior and ethics

Is fair and consistent in all matters

Is compassionate without being ineffectual

Teaching Experience (Preferred) Documents prior teaching experience, college-level

preferred

Presents syllabi from previous courses if possible

Industry Experience (Preferred) Documents current industry experience

Exhibits credible content knowledge

Holds continuing professional membership(s)

Documents ongoing professional development

Demonstrates knowledge of subject area and


26/36


applications

Relates industry experience to classroom


27/36


JOB DESCRIPTIONS:Academic Affairs

Title: FACULTY, Part time/Adjunct (term-to-term employees)

Department: Academic Affairs

Position Summary:

Instruct and facilitate meaningful learning of the course competencies in the curriculum and

proactively support all facets of the learning environment. Provide education through learning-

centered instruction that will enable graduates to fulfill the evolving needs of the marketplace.

Encourage a culture of learning that values mutual responsibility and respect, life-long learning

and ethics as well as personal and professional development.

Key Job Elements:

Provides competency-based education

Designs/delivers class instruction

Enables student exit competencies

Delivers learning-centered instruction

Promotes student success

Manages the learning environment

Contributes to a culture of learning


28/36


Relates professional/life/industry experience to learning

Organizational Relationships:

Reports to: Dean of the School of, Department Chair/Academic Department Director

VP/Dean of Academic Affairs

Collaborates with: Academic Affairs Department

Other school/campus functional areas

Curriculum Task Forces and other school committees

Competencies and Expectations:

Key Job Element ACCOUNTABILITY PERFORMANCE EXPECTATION

Provides

Competency-

Based Education

Graduate Outcomes

Persistence Rate

Employment Rate

Starting Salaries

Provides instruction which creates the

opportunity for the student to achieve

outcomes

Aligns with the EDMC models for

curricula

Supports the EDMC style of system

delivery


29/36


Designs/Delivers

Class instruction

Syllabus/Lesson Plans

Classroom Assessment

Develops instructional plans to meet course

competencies

Develops activities which support lesson

objectives

Delivers online instruction as approved

(online instructors only)

Enables Exit

Competencies

Student Achievement

Evaluation of Learning

Provides instruction which fosters student

achievement of identified exit competencies

Establishes/communicates student

performance criteria and evaluation based

on exit competencies

Delivers

Learning-

Centered

Instruction

Active Learning

Student-Oriented

Establishes a classroom environment

conducive to collaborative learning and

active student involvement

Prepares for classes to ensure student

success


30/36


Demonstrates effective planning and

preparation

Exhibits flexibility in style and work

schedule

Promotes Student

Success

Persistence

Student Satisfaction

Helps students solve problems that may

impede successful program completion

through advising, tutoring, mentoring,

and/or referring students on academic or

personal matters

Participates in student persistence activities

Exhibits passion for teaching and students

Engages students in the learning process

Meets students as individuals rather than a

group

Thinks outside traditional academic

processes to bring information to students

Champions student concerns as they relate

to student success


31/36


Participates in activities that build the

reputation of the school

Manages the

Learning

Environment

External/internal

Audits

(including

accreditation, Title IV

and other regulatory

compliance visits)

Keeps accurate records

Submits grade and other reports on time

Enforces school/campus academic and

attendance policies

Demonstrates management skills in the

classroom

Demonstrates lesson management in the

classroom

Maintains classroom in which all students

are treated in an equitable manner

Contributes to a

Culture of

Learning

Personal Development

Service

Supports local school/campus or online

events such as Open House, orientation,

graduation, etc. as appropriate

Participates in departmental meetings,


32/36


workshops, committee sessions, activities

and initiatives as appropriate

Relates

Professional/Life/

Industry

Experience to

Learning

Experiential

Knowledge

Continues to develop professional/technical

skills

Introduces industry perspective into course

materials as appropriate

Maintains active awareness of

professional/industry trends and

opportunities

Position Requirements/Preferences

Appropriate Credential(s) Has sufficient credits in major area

Meets accreditation criteria for major area

(note: Faculty are required to seek higher credentials

when so instructed; ex. school changes accreditation

status)


33/36


Organizational Skills Handles and meets multiple deadlines

Develops projects without continuing direct

supervision

Reports necessary data and information timely and

accurately

Professional Criteria Demonstrates excellent written and oral

communications skills

Communicates in Standard English with correct

usage and a professional vocabulary

Is able to develop a professional rapport with diverse

school/campus constituents

Demonstrates good conflict resolution skills

Learning-Centered Demonstrates student-focused style

Exhibits willingness to learn from students

participation

Learning Styles and Methodology Demonstrates awareness and sensitivity to student


34/36


needs and learning styles

Presents program content in multiple modalities to

enhance learning

Delivers subject material in ways that engage students

Maintains the quality of education while presenting

difficult concepts to learners

Customer-Orientation Responds to all constituents in a timely, civil, and

courteous manner

General Characteristics Relates to all functional areas within the school

Works collegially with staff and faculty

Has an attitude of service to the institution and its

constituents

Is self-motivated, articulate, and professional

Exhibits a positive, enthusiastic, and constructive

attitude while dealing with issues

Works well with other faculty and staff within the

school and community

Possesses a high energy level


35/36


Is able to see beyond his/her courses to the vision of

the school as a whole

Considers him/herself a part of the Academic Affairs

team rather than a single faculty member

Personal Characteristics Enjoys learning

Exhibits professional behavior and ethics

Is fair and consistent in all matters

Is compassionate without being ineffectual

Teaching Experience (Preferred) Documents prior teaching experience, college-level

preferred

Presents syllabi from previous courses if possible

Industry Experience (Preferred) Documents current industry experience

Exhibits credible content knowledge

Holds continuing professional membership(s)

Documents ongoing professional development

Demonstrates knowledge of subject area and

applications

Relates industry experience to classroom


36/36
