Modeling Assessments of Innovative Physics

Courses

A Symposium

Institution’s Educational Outcomes

To prepare students who…

possess a breadth of integrated, fundamental knowledge in basic physics

build a depth of knowledge in exploring the contradictions between personal assumptions and evidence from the physical world.

can frame and resolve ill-defined problems can communicate effectively are independent learners can work effectively with others are intellectually curious can apply knowledge and skills to the

unique tasks of a profession

Physics Department’s Curriculum Goal

to develop introductory curriculum which exploits the use of new technology in order to promote the development of critical thinking and problem solving skills, thus enhancing career preparation.

ZX Approach to Teaching Physics

required inventive problem solving that addressed complex situations which had

ill-defined approaches and unclear solutions

deluged students with relevant and irrelevant information and

required analysis, criticism, and synthesis

ZX Approach to Teaching Physics

modeled conditions met by human beings in real world vs. working within an abstracted or generalized framework

encouraged scholarly research using the CD-ROM resources

planned for individualized problem-solving required learner creation of problems to be

solved by peers.

Policy Support for ZX Approach

Part of the education students receive is responding to stress and allocating precious time to a variety of demands. Responding to the demands of coursework is one of the areas in which [students] must often make critical decisions. While the faculty should understand that [students] have many things to do in a given day, this level of activity no way justifies diminishing the rigor of our courses. [Students] must also learn responsibility, and we must allow them the chance to fail as well as succeed.

Faculty Handbook, p. 30.

Action Research OverviewPurpose

• To “track” the impact of innovation in the approach to teaching physics for a semester:– Implementation of CD-ROM resource– Use of e-technology as major instructional

tool– Attitudes of instructors and students– Use of ill-defined problems as central

instructional approach

Multiple Methods ElementsQuantitative Data Collected

Pre- and Post-data

Multiple “aptitude” entry measures (Pre-)Nature of Science Profile (Pre- Post-)Strategies in Teaching Physics (Pre- Post-)Attitude to Science (Physics) (Pre- Post-)Force Concept Inventory (Pre- Post)Common summative examination (Post-)

Multiple Methods ElementsQualitative Data Collected

3 student interview sessions

Weekly email student diaries

Instructional staff interviews and notes on change process

Theoretical FrameworksAdult Intellectual Development

• William Perry (1970, 1981)• Dualism -- instructor and textbook is authority

• Multiplicity – everyone has right to an opinion

• Relativism – analytical thinking skills emerge;

ability to critique own ideas and those of others

• Validity – not all ideas/positions are equally valid

Theoretical FrameworksExpert vs. Novice

– Ability to chunk info into familiar patterns

– Knowledge organized around core concepts or big ideas

– Knowledge reflects contexts of applicability s

– Automaticity in retrieving relevant characteristics for problem solving

– A need to possess content pedagogical knowledge

– Mental flexibility, a positive view of one’s own accomplishments and a belief that there is much more to learn.

Experts have….

Theoretical FrameworksRIGOR DEFINED

• Rigor is the goal of helping students develop the capacity to understand content that is complex, ambiguous, provocative, and personally or emotionally challenging.

Strong. R., H. Silver and M. Perini. (2001) Teaching What Matters Most: Standards and Strategies for Raising Student Achievement. Alexandria, VA: Association of Supervision and Curriculum Development. P. 7.

Interview Protocol

• 3 instructors on experimental team• 12 students, 6 from each experimental course• Selected randomly, ensuring gender balance• 3 interviews

– Entry level – first week of semester– Mid-point– Exit level – 2 weeks before final exam

Feedback from interviews given to instructors

Interview Cohort12 students from experimental group

Reasons for volunteering…. 9 use of computers to learn 7 CD-ROM instead of textbook 3 no books to carry 1 many new resources available 1 a new opportunity 1 can explore Physics deeper

Interview CohortPre-study interviews

Previous Physics experience…

4 one year High School 3 two years High School 2 one year advanced level physics 1 one year College plus 1 year High School 2 none

Interview CohortPre-study interviews

Subjects described learning style…

5 as visual3 as hands-on2 as problem-solvers2 need an interesting instructor

Interview CohortPre-study interviews

Primary subject interest…

5 Math and Science 1 Science 2 Math 1 Computer Science 1 History 1 Political Science 1 Spanish

Least favorite subject…

6 English 3 Writing 2 History 1 Math

LESLIE

Interview CohortPost-study interviews

Reported experiences and attitudes…

Appreciated…

working with others toward a common goal

using technology as a tool for studying Physics

Interview CohortPost-study interviews

Reported experiences and attitudes…

Concerns…

negative impact of time constraintsconfusion in choice-making activitiesconcern about adequate preparation

for final departmental assessment

Quantitative Research Review:Instruments Used

Strategies in Teaching Physics (SITP) Pre-PostNature of Science Profile (NOSP) Pre-PostAttitude to Science (Physics) (ATP) Pre-PostForce Concept Inventory (FCI) Pre-PostMultiple “aptitude” entry measures: Pre-Common summative final exam: Post-

Why Use Surveys?Some of the Research Questions

What are the students’ and instructors’ approaches to the teaching and learning (curriculum) context of physics? (SITP Questionnaire)

What are the students’ and instructors’ approaches to the philosophical context of science (physics)? (NOSP Questionnaire)

What are the students’ and instructors’ attitudes to social and educational contexts of physics - including the use of electronic technology? (ATP Questionnaire)

Strategies in Teaching Physics(SITP) Pre- Post-

Administered to Instructors and Students Pre-post80 items - 3 subtests - 6 point scale

SITP: Examines Three Curriculum Models

Sub-test 1: Neo-classicism (reproduction) (memorize and recall physics facts, concepts and processes)

Sub-test 2: Liberal-progressivism (reconstruction)(active personal construction of physics knowledge)

Sub-test 3: Social-criticism (interactive reconstruction)(students participate as co-learners of physics)

ONE TEACHING-LEARNING CONTEXT INSTRUCTORS’ RESULT (SITP)

Four instructors exhibited noticeable differences in preferred teaching strategies.

Nature of Science Profile (NOSP)Instructors & Students: Pre- Post-

24-item, 11 point Likert scale, 5 subtests

Examines the philosophical context of science (physics) along five axes:

Relativism < -- > Positivism (RPP axis)

Inductivism < -- > Deductivism (IDP axis)

Contextualism < -- > Decontextualism (CDP axis)

Process < -- > Content (PCP axis)

Instrumentalism < -- > Realism (IRP axis)

ONE PHILOSOPHICAL CONTEXT INSTRUCTORS’ RESULTS (NOSP)

INSTRUCTOR NATURE OF SCIENCE PROFILES

-120

-100

-80

-60

-40

-20

0

20

40

60

RPP IDP CDP PCP IRP

NATURE OF SCIENCE SUB-TESTS

SU

B-T

ES

T-S

CO

RE

S (

PE

RC

EN

T)

Instr 1

Instr 2

Instr 3

Instr 4

For the Relativist-Positivist subtest (RPP), the more negative the score, the stronger the Relativist position. The higher the positive score, the stronger the Positivist position – similarly for IDP, CDP, PCP & IRP.

The NOSP graph shows strongly differing ideas amongst the instructors about the philosophical context of science (physics).

Such philosophical differences about the nature of science (physics) are NOT INCONSEQUENTIAL.

Such fundamental differences of philosophy undercut attempts at curriculum reform.

Conflicts of Instructors’ Ideas about the Nature of Science

(Physics) Context

Comparing SITP Curriculum Contexts with the NOSP Philosophy of Science Contexts

Neo-classicism (NC) vs Liberal-progressivism (LP) (knowledge reproduction) (knowledge construction)

Realism <-> Instrumentalism (IRP)

Inductivism <-> Deductivism (IDP)

Positivism <-> Relativism (RPP)

Decontextualism <-> Contextualism (CDP)

Content <-> Process (PCP)

Social Criticism is not represented in the table above because the philosophical differences between LP & SC are far more subtle than between NC & LP (or NC & SC).

Subtests: Social implications of physics: SP Attitude to scientific inquiry in physics: AP Attitude to teaching/learning physics: TP Physics as a personal threat: PTP Physics as a practical hands-on activity: HP Socio-cultural value of physics: VP E-technology in learning/teaching physics: ET

Attitude to Physics and E-Technology(ATP) – 1

87 questions - 7 Subtests - 7 point scale

The bar graph shows instructors’ variability. Note SP; AP; HP & VP.

INSTRUCTOR ATTITUDE TO PHYSICS SUB-TESTS

0

5

10

15

20

25

30

35

40

45

SP AP TP PTP HP VP

SU

B-T

ES

T S

CO

RE

S (

PE

RC

EN

T)

Instr 1

Instr 2

Instr 3

Instr 4

Attitude to E-Technology (ATP) – 2; 27 Extra Items; 4 sub-tests

Created specifically for the research study at the request of the focus group:

Interest in E-technology (7 ZIP items)Useful aid to study (7 ZUP items)Aid to Multi-tasking (6 ZMP items)Personal Control (7 ZCP items)

Instructors do not exhibit a uniformly high response. Some head in opposite directions.

INSTRUCTOR ATTITUDE TO E-TECHNOLOGY IN PHYSICS TEACHING

-10

0

10

20

30

40

50

60

ZIP ZUP ZMP ZCP

E-TECHNOLOGY SUB-TESTS FOR INSTRUCTORS 1, 2, 3, 4

SU

B-T

ES

T S

CO

RE

S P

ER

CE

NT

1

2

3

4

Action Research Summary: 1 Instructors’ Quantitative Result

If a Physics Department is genuinely interested in developing a coherent and effective approach to collaborative teaching and shared instructional goals, then the skills, attitudes, and dispositions of the instructors must be examined and taken into account in the process of curriculum reform.

Students’ Quantitative Result 1Aptitude Pre-tests c.f. Final Exam

7 Aptitude Predictors Employed:• 2 Standardised Achievement Tests • Math Aptitude; Algebra Aptitude; Calculus Aptitude• Chemistry Aptitude; Physics Aptitude• ACCOM Score (Academic Composite of Aptitudes) –

used to allocate students to Honours Classes.

THE ONLY PREDICTOR TO PRODUCE A SIGNIFICANT ASSOCIATION WITH FINAL PHYSICS EXAMINATION RESULTS WAS THE CHEMISTRY APTITUDE TEST. WHY?

Some Little-known Facts About:Neo-classical Instruction and Examinations.

The “Neo-classical Instruction Effect”: Teaching procedure: “Equal exposure”/”Equal rate” Student progress: By differential entry aptitude/ability rate –

“Slow” students progress slower (differentially) by aptitude…“Fast” students progress faster (differentially) by aptitude.

Student “performance” order: Entry “aptitude” order should be THE SAME AS exit summative exam order (except for minor “accidental” influences – poor teaching, illness, lack of motivation, etc)

Student performance distribution: Remains “normative” but with a differentially greater “spread”/variance (caused by differential “spreading” of rates of progress by aptitude).

Grading: Could be done “fairly” and “equally” at the beginning of semester using valid aptitude/ability test order (theoretically).

• If you are a neo-classical teacher, just for “fun”, compare your pre-course aptitude order with your post-course final exam order using Spearman rho!!

• You have a problem somewhere if there is a significant difference.

Students’ Pre-Post Quantitative Result 2Force Concept Inventory

Hº: “No difference amongst average scores by treatment groups”: RETAINED (Kruskal-Wallis One-

way Anova test)

If the FCI is a valid and reliable independent test of understanding of Newtonian mechanics, then:

What did the physics course instruction teach the students about Newtonian Mechanics during the semester?

Students’ Pre-Post Quantitative Result 3Strategies in Teaching Physics (SITP)

Neo-classical Sub-test (NCP):Pre-Post Preference Scores: DECLINED

Liberal-progressive Sub-test (LPP)Pre-Post Preference Scores: NO DIFFERENCE

Socially-critical Sub-test (SCP)Pre-Post Preference Scores: NO DIFFERENCE

Further: No difference by gender, class grouping, or instructor.

Students’ Quantitative Result 4 (NOSP) Favoured Ideas Profile

Relativism<->Positivism (RPP): Undiscriminated Inductivism<->Deductivism (IDP): UndiscriminatedContextualism<->Decontextualism (CDP):

DecontextualismProcess<->Content (PCP): Process Instrumentalism<-> Realism: (IRP): Realism

Further: No difference by gender, class grouping, or instructor.

Students’ Pre-Post Quantitative Result 5Nature of Science (Physics) (NOSP)

Relativism<->Positivism (RPP): No difference Inductivism<->Deductivism (IDP): No differenceContextualism<->Decontextualism (CDP): No

differenceProcess<->Content (PCP): No difference Instrumentalism<-> Realism: (IRP): No difference

CONCLUSION: Students retained the ideas formed in High School. Physics 110 had no effect on students’ views about the nature of science (physics).

Students’ Pre-Post Quantitative Result 6Attitude to Physics (ATP)

Pre-Post Sub-test Results: Social implications of physics (SP): Significantly Lower Attitude to scientific inquiry in physics (AP): Significantly

Lower Attitude to teaching/learning physics (TP): No difference Physics as a personal threat (PTP): Significantly Lower Physics as a practical hands-on activity (HP): No difference Socio-cultural value of physics (VP): Significantly Lower

Further: All pre-post attitudes had positive support. No pre-post difference by gender. There were significant differences by class

section and instructor.

Students’ Pre-Post Quantitative Result 7 (ATP) Attitude to E-Technology

Pre-Post Sub-test Results: Interest in E-technology (ZIP): Significantly lower Useful aid to study (ZUP): Significantly lower Aid to Multi-tasking (ZMP): Significantly lower Personal Control/Vulnerability (ZCP):

Significantly lower

Further: Female ZIP, ZMP & ZCP post-tests were significantly lower. Section grouping post-tests were all significantly lower. Instructor grouping post-tests were significantly lower for ZIP only.

Students’ Final Summative Written Exam Results - 8

Overview of Final Summative Exam Results: There were no significant differences amongst Final

Exam scores by experimental section, by instructor, or by treatment group.

In general, students were not disadvantaged through participation in the CD-ROM project.

It is not possible to make any quantitative statement about academic gains by the CD-ROM project groups through the use of the CD-ROM.

In considering the total student population, Final Exam results for females were significantly lower than for males.

Some of the Conclusions and Recommendations 1

No pre-course aptitude measures were valid outcome predictors – except the chem aptitude test.

There is no value in using the FCI with the current physics curriculum and final examination.

The current teaching and final examination system discriminates against females (shown by pre-post FCI results and course entry aptitude measures).

There were no differences in final performance across sections, instructors, or treatment group.

Some of the Conclusions and Recommendations 2

There was a significant drop in students’pre-post dispositions to the neo-classical teaching model.

Dispositions to liberal-progressive and socially-critical models of curriculum and instruction remained stable.

There were no pre-post changes to sub-test dispositions by gender, section grouping or instructor.

Both instructors and students may be ill-prepared for the teaching of physics using inquiry procedures to support critical thinking and problem-solving approaches directly related to the institutional objectives.

Instructors could benefit from workshops involving history and philosophy of science, curriculum theory, educational psychology, and educational measurement.

Some of the Conclusions and Recommendations 3

Four of six ATP attitudes declined significantly across the semester. Section attitudes converged so that all sections became more alike.

Students’ E-technology attitudes declined across the semester. Some attitudes remained more positive than others. Sections by instructor attitudes converged to become more alike. Reasonable support remained for E-technology.

The Physics 110 programme made no impact on the sophistication of students’ metaphysical viewpoints about the nature of science (physics).

Physics 110 students’ attitudes tend NOT to support inquiry, critical thinking and complex problem-solving teaching and learning procedures.

Some of the Conclusions and Recommendations 4

“Immersion in”, “exposure to”, and “coverage of” will NOT be sufficient to attain the departmental and institutional goals.

Instructors most aligned with departmental and institutional goals are those who receive the least favourable end-of-semester evaluation of teaching .

Students need to be re-educated to think differently about what it means to learn, understand, do, and be competent in physics.

Physics 110 students misunderstand, disagree, oppose, and show discontent with teaching that is not seen to focus directly upon final examination specifications.

Some of the Conclusions and Recommendations 5

The Physics 110 curriculum and instruction model must be reviewed in the light of the last 30 years of research on the intellectual development of young adults.

The Physics 110 curriculum and instruction model must respond to the cognitive agility, flexibility, and innovation required by the ambiguity, complexity and change features of the 21st century world.

Students need to be re-educated to think differently about what it means to learn, understand, do, and be competent in physics and be moved from a passive to an active responsibility for their learning and understanding.

The Physics Department must re-orient its approach to academic “rigour” so that the term no longer implies the brain is a muscle that must be subjected repeatedly to severity, hardship, adversity and stress to improve its functioning and performance.

Features of Action Research Aim: • Create and study change.

Minimal requirements:

• Social practice (not content) is the research subject.• Proceed systematically in repeated cycles of observing,

planning, doing, and reflecting.• Involve those responsible for current practice and future change. • Collaborate with all stakeholders through appropriate

representation.

Useful words:

• Re-examine; reinvent; reconceptualize; re-imagine; refine; refresh; reform; rebuild; reconstruct; revise; remodel; regenerate; revisit.

Action Research ProceduresProcedure: (What we did.)

Form a reference group. Identify or define the problem(s) or issue(s). Identify clearly what happens now. Reflect and refocus on how and what might, should, and can be done differently.

• Seek disagreement.

• Redefine the problem as necessary. Settle on the research question(s) Develop a strategy for collecting data, solving the problem or implementing an idea. Use multiple flexible methods. Collect data, implement action, problem-solve, test ideas. Observe, evaluate, conclude – study the consequences of actions, specify learnings,

make sense of experience, describe, explain. Ask, “So what?” and “What next?”

• Continue with ongoing action research cycles to make further improvements. Share your learning with others.

What We Learned….Still requires work..

What were strengths of study?Institutional, departmental, staff, and student involvement.Cooperative extensive participation of staff and studentsExtensive data gathering was possible.

Non-threateningNon-invasiveNon-personalThoroughUnpleasant or unexpected findings were fairly reportedRecommendations were made without fear or favourWhat was missing?

Impact of time and budget?Influence of institutional culture?Influence on instructors’ risks?

Responder Frits Gravenberch

• 20 minutes

Question and AnswerSession

• 15 minutes