Research on Authentic Assessment Using a Virtual World for Learning Geology

Bernhardt Saini-Eidukat1, Donald Schwert1, Brian Slator2, Lisa Daniels3, and Jeffrey Terpstra4

1Dept. of Geosciences; 2Dept. of Computer Science; 3School of Education; 4Department of Statistics

North Dakota State University, Fargo, ND

NDSU WWWICWorld Wide Web Instructional Committee

Paul Juell

Donald SchwertPhilip McClean

Brian SlatorBernhardt Saini-Eidukat

Alan WhiteJeff Clark

Lisa DanielsJeff Terpstra

WWWIC faculty supported by large teams of undergraduate and graduate students

WWWIC’s virtual worlds research supported by NSF grants DUE-9752548, EAR-9809761, DUE-9981094, ITR-0086142

and EPSCoR 99-77788, and Department of Education FIPSE P116B011528

The Geology Explorer Project

Educational Game designed to provide authentic learn-by-doing experience

•Exploration of a spatially oriented virtual world•Practical, field oriented, expedition planning and

decision making•Scientific problem solving (i.e., a “hands on”

approach to the scientific method)

Balancing Pedagogy with Play

Games have the capacity to engage!

• Powerful mechanisms for instruction

• Illustrate real-world content and structure

• Promote strategic maturity (“learning not the law, but learning to think like a lawyer”)

Technical Approach

• Networked, internet based, client-server simulation

• UNIX-based MOO (Multi-User Dungeon, Object Oriented)

• Java-based clients (text and graphical versions)

The Setting

•Planet Oit - similar to Earth, but opposite the Sun

•Students “land” on Oit to undertake exploration

•Authentic Geoscience goals - e.g., to locate, identify, and report valuable minerals

The Simulation

~50 places: desert, cutbank, cave, etc.

~100 different rocks and minerals

~15 field instruments: rock pick, acid bottle, magnet, etc.

~Software Tutors: agents for equipment, exploration, and deduction

Graphical Navigation

Database Map of Planet Oit

Carrying out Investigations:

Players Identify Outcrops Using Tests

Results of Tests (in this

case, acid reactivity)

Detail Images

Carrying out Investigations:Players Identify Outcrops Using Tests

Because each mineral in the rock can be distinguished, the test can be performed on each mineral. Each returns whether it is acid reactive or not.

Coarse-grained rocks

If the rock contains more than 10% calcite, it fizzes actively when acid is poured on it. If the rock contains between 0 and 10 percent calcite, its powderized form will fizz. Also, if the rock contains more than 25% dolomite, its powder will fizz actively. Finally, if the rock contains no calcite or dolomite, it will not fizz.

Fine-grained rocks

If the property acid_reactive is 2, the mineral fizzes actively when acid is poured on it. If the property acid_reactive is 1, and a powder is made of the mineral, the powder will fizz actively. If the acid_reactive property is 0, the mineral does not fizz.

Minerals

Most other objects in the game have a acid_reactive property of 0. Pouring acid on them will cause them not to fizz. Players are an exception to this rule. You are unable to pour acid on other players.

Other objects

Example of Programming Actions:

Acid Bottle: "pour acid bottle on substance"

Deductive Tutors

• Equipment tutor

• Exploration tutor

• Science tutor

Detects when a student makes a wrong guess and why (i.e. what evidence they are lacking); or when a student makes a correct guess with insufficient evidence (i.e. a lucky guess)

Intelligent Software Tutoring Agents

Students/players interact with the intelligent tutoring agents

Players can ignore advice and carry on at their own risk

Virtual Mapping

SS

LS

SS

Schist

Marble

Qzite

Schist

QtziteDike

Schist Gneiss

LSMarble

Qzite

Qzite

SS

SS Schematic map - not to scale

• Locate yourself and outcrops using topo maps and aerial photos

• General rock identification

• Create a Geologic Map

Virtual Mapping

Tutorials on Concepts of Geologic Mapping are Available in the Environment

A Player (appearance can be changed)

Virtual Mapping

Base Maps for Adding Data and Creating a Geologic Map

“aerial photo”

Virtual Mapping

Base Maps for Adding Data and Creating a Geologic Map

Topographic map

Virtual Mapping

Outcrop Locations are Shown with Markers

Rock Type Markers Show Locations of

Identified Outcrops

You Are Here

Player Creates a Geologic Map Based on Outcrop Locations, and Can Get Immediate Feedback

Virtual Mapping

Virtual Mapping:Formative Assessment

Virtual Mapping:Assessment

Computer Scoring of Your

Map

Rejects the notion of standardized multiple choice tests

Pre-game narrative-based survey• short problem-solving stories • students record their impressions and questions

Similar post-game survey with different but analogous scenarios

Surveys analyzed for improvement in problem-solving

Assessment Using Scenarios: Summative

You are in northern Manitoba on a fishing trip. Fishing has been good. At dawn on the day of your departure you cut across country to a remote bay and have more good luck: you catch your limit of walleye!

Assessment Using Scenarios

On the way back to the lodge, you stumble across a heavy, moss-covered rock on the shoreline, flipping it over. Looking down, you see the underside shines with a brilliant metallic yellow. You pick up the rock and lug it back to the lodge with you.

Assessment Using Scenarios

When it’s time to leave, the pilot notes that you are 45 pounds overweight...exactly the weight of the rock you found. She says, "At $8 a pound, eh, you owe $360."

• As a geologist, what do you do?• List the things you would consider in your decision.• List the questions you would ask yourself, and give

the reasons behind those questions.

Assessment Using Scenarios

Assessment MethodGeneral Problem Solving (60 pts)

1.1 Form a Hypothesis: 20 pts

1.2 Suitable Test: 20 pts

Decide to Test Experimental Design, Comparison, Etc. First Specific Test Second Specific Test 1.3 Conclusions based on evidence: 20 pts

Stated and correct with correct explanation Stated and correct Stated and correct with incorrect explanation Implied and correct Stated and incorrect Just implied Implied and incorrect

?

Specific Information (24 pts)

2.1 Color/Luster: 6 ptsMention testing for Describe how Specific properties

2.2 Malleability/Hardness: 6 pts(ditto) 2.3 Specific Gravity: 6 pts(ditto)

2.4 Other Test : 6 pts(ditto)

Assessment Method

Assessment Method

Expert Knowledge (16)

3.1 Current price of gold: 5 ptsHigh knowledge Some knowledge No knowledge  3.2 Specific Geologic Conditions : 5 pts(ditto)

3.3 Judges prerogative: 6 pts

??

Validation: Multiple Graders Assessment Protocol

Pearson Product Moment Correlation Coefficent between graders

Grader 1 Grader 2 Sample Size Coefficient

jennifert arico 263 0.747

kima arico 263 0.849

kima jennifert 263 0.734

2003 Post Test

0

1020

30

40

5060

70

8090

100

0 20 40 60 80 100

Aric

Kim

A a

nd

Jen

n T

jenn post

kim-a post

Current Assessment Research: Scenario Order; Team vs. Solo play

Students Scen 9/10 Scen 10/9 TotalTeams of 2 19 20 39Solo 162 155 317Total 181 175 356

Preliminary Data from 2004 – scoring is ongoing!

Number of Observations Used Here: 190

Conclusions: Scenario Order; Team vs. Solo play

Scenario Combination Solo Team P-val SignifScen 9/10 9.00 6.83 0.6637 NSScen 10/9 -0.16 11.31 0.0183 S

Net Improvement for 2004

The response variable is Net Improvement based on one grader, “Kelly”.

The data were analyzed as a two-factor factorial design with interaction, based on ordinary least squares methodology.

Geology Explorer Development Team Graphics • Acey Olson (Rendered Landscapes)• Rebecca Potter (Rendered Images, Icons, and Landscapes)• Dan Reetz (Rendered Images, 3D)• Shannon Tomac (Rendered Landscapes, Icons, & Images, Comic Book)• Daniel Turany (Rock/Mineral Photographs)Staff • John Bauer; Java Client (GUMI-bare and GUMI-game) Design and Implementation• Nem Schlecht• Mark Tinguely • Brad Vender; MOO-side HTML server

Students Graduate Research Assistants:

•Kim Addicott; Assessment•Otto Borchert; Software tutors, Java client development •Bob Cosmano; Makes Java Go; Client Implementation•Krista Dauner; Tutoring Scripts •Rahul Devabhaktuni; Hydrology (Water) on Oit •Kuo-Di Jian; Java Tool Integration •Yongxin "George" Jia; Java/LambdaMOO Hierarchy Tool •Guy Hokanson; Software tutors, dynamic HTML •Bhaskar Marthi; Instrument verbs •Vidyalatha Nagareddy; Java/LambdaMOO Entity Tool •Shanhong Wu; Java3D

Undergraduate Research and Independent Study Students: •Bryan Bandli; Assessment and Evaluation •Chris Fish; Endgame Design and Implementation •Kelly Gorz; Assessment and Evaluation•Julia Karst-Grey; Geologic Content and Assessment •Lucas Koehntop; Java consulting •Ned Kruger; Assessment and Evaluation •Acey Olson; Bryce Landscapes, Tutoring, Player History, Interface Design •Carson Rittel; Content Quality Control, Help Database •Joy Turnbull; Content Quality Control, Help Database•Dean Vestal; Content Quality Control, Help Database, Interface Design •Mindy Sue Vogel; Geologic & Quality evaluation, Alternative exercises •K.C. Vorthmann; Geologic Content and Assessment•Jeff Walsh; Geologic Content and Assessment•Jane Willenbring; Assessment and Evaluation

Acknowledgments:Mark Tinguely, who saved our world when its universe imploded, and Dave Schmidt for the name: Planet Oit.

•Aric Olsen; Geologic Content and Assessment•Jennifer Thorstad; Geologic Content and Assessment

http://oit.ndsu.edu

World Wide Web Instructional Committee (WWWIC)

North Dakota State UniversityFargo ND