Charting and Measuring Changes in students’ Knowledge of a Curriculum

TopicRoss J Todd

Center for International Scholarship in School Libraries

Rutgers, The State University of New Jersey

New Brunswick NJ USA

rtodd@scils.rutgers.edu

www.cissl.scils.rutgers.edu

Acknowledgements

“Impact of School Libraries on Student Learning”

Funded by the Institute for Museum and Library Services (IMLS) USA 2003-2005

Dr Carol Kuhlthau, Co-Principal Investigator

Dr Jannica Heinström, Research Associate CISSL

Nora Bird, Doctoral Student, Project Manager

Project GoalsConceptual: To understand how students build on their existing knowledge of a topic and how their knowledge of a topic changes;

The transformation and integration of found information into existing knowledge, and the creation of new knowledge.

Methodological: To test approaches to the elicitation and representation of new knowledge in order to document changes to knowledge.

Professional: To develop school-based tools for measuring and charting learning: SLIM Toolkit

Information UsePhysical and mental acts employed by humans to incorporate found information into their knowledge base or knowledge structure (Wilson, 1981, 2000).

Information use research is the least developed in the information behavior research Vakkari (1997); Spink & Cole (2005).

Given that school students typically spend a lot of time in the library researching curriculum assignments:

do they actually learn anything new when they seek and use information sources?

And if they do, what does this learning look like, in terms of new knowledge about a curriculum topic?

How does their existing knowledge of a topic change through the task?

How can this change of knowledge of a curriculum topic be measured?

Knowledge Representation: Common Assumptions

Subjective, private knowledge is structured, and can be adequately elicited and represented (Rumelhart & Norman, 1985, Markman, 2002);

Knowledge may be depicted as the overall structure of concepts linked together relationally (Ausubel, 1963; Ausubel, Novak & Hanesian, 1978; Brookes, 1975);

Past experience and prior knowledge form the basis for constructing new knowledge (Bartlett, 1932; Dewey, 1938; Vygotsky, 1962; Kelly, 1963);

Previous knowledge forms a filter in the selection and rejection of new information, and as a result of encountering new information, knowledge structures may or may not change (Brookes, 1980; Chinn & Brewer, 1998);

Conceptual change tends to develop gradually by modifying, refining and transforming previous knowledge structures (Kintsch & Vab Dijk, 1978; Dole & Sinatra, 1998; Todd, 1999b).

Gaps in our Understanding

Mapping the actual knowledge output as students progress through the stages of information searching has been given limited attention.

Need to develop more accurate techniques for representing and measuring conceptual structures and how they change (Pennanen & Vakkari, 2003)

Knowledge Representation & Measurementin School Setting

Use of verbal and written protocols such as essays, projects, exams, and presentations.

Amount and nature of knowledge of a topic is determined primarily in classroom settings by subject experts (teachers) who match the nature of ideas to some expected target or expert conception (typically curriculum content requirements), and within a prescribed word / page count

Little measurement of change of knowledge in terms of initial existing knowledge

Measuring Knowledge through Relations and Structure

Studies by Chi (1992), Chi & Koeske (1983), and Gobbo & Chi (1986)

Measure changes in knowledge structures in terms of the number of links, the nature of clusters of statements and the characteristics of coherence and structural centrality.

AMOUNT OF KNOWLEDGE STRUCTURE OF KNOWLEDGE

Measuring Students’ Change in Knowledge:Research Questions

Substance of Knowledge: What changes in content of relational statements are evident in students’ knowledge descriptions as they undertake guided inquiry project?

Structure of Knowledge: What changes in structure of relational statements are evident in students’ knowledge descriptions as they undertake guided inquiry project?

Amount of Knowledge: What changes in amount of relational statements are evident in students’ knowledge descriptions as they undertake guided inquiry project?

Extent of Knowledge: What changes in perceptions of their knowledge are evident in students’ knowledge descriptions as they undertake guided inquiry project?

Title of Knowledge: How do the students’ titles given to their topics change as they undertake guided inquiry project?

Schools Context & Sample10 New Jersey public schools

Experienced and expert school librarians

Diverse public schools

10 school librarians working on curriculum projects with 17 classroom teachers

574 students in Grades 6 – 12; range of disciplines

Inquiry Training Institute Feb 24, 2004: overview and critique of units, use of data collection instruments, procedures and ethical guidelines

Data Collection Instruments

Five data collection instruments were used to collect the data from the students:

1. Writing Task 1 (at initiation of inquiry unit)2. Writing Task 2 (at midpoint of inquiry unit) 3. Writing Task 3 (at conclusion of inquiry unit)4. Search Journal Log5. My Research Style

The instruments consisted of a combination of qualitative and quantitative questions.

Writing Tasks

Writing task 1 and 2 consisted of the following questions

1. Write the title that best describes your research project at this time.

2. Take some time to think about your research topic. Now write down what you know about this topic.

3. What interests you about this topic?4. How much do you know about this topic? Check () one box

that best matches how much you know. Nothing, Not much, Some, Quite a bit and A great deal

5. Write down what you think is EASY about researching your topic.

6. Write down what you think is DIFFICULT about researching your topic.

7. Write down how you are FEELING now about your project. Check () only the boxes that apply to you. Confident, Disappointed, Relieved, Frustrated, Confused, Optimistic, Uncertain, Satisfied, Anxious or Other.

Additional Questions at Writing Task 3

1. What did you learn in doing this research project? (This might be about your topic, or new things you can do, or learn about yourself)

2. How did the SCHOOL LIBRARIAN help you?

3. How did the TEACHER help you?

Substance of knowledge

Classification of Statements: based on nature of relationships between concepts

Graesser & Clark (1985) Structures and procedures of implicit knowledge. Norwood, N.J.: Ablex.

Properties: statements describing characteristicsManner: statements describing processes, styles, actionsReason: statements of explanations of how and whyOutcome: statements providing end resultCausality: statements showing some event causally leads to anotherSet Membership: statements about class inclusionImplication: statements showing predictive relations,

inference, implied meaningValue judgment: statements presenting personal position or viewpoint

Classification of Statements

Properties: The color of Valentine’s day is redManner: People drive aggressively in USAReason: The wall was constructed to block invadersOutcome: (People eat too much) As a result, people

got very sick Causality: Too much alcohol can lead to liver failureSet Membership: Michelangelo created works such as statue of David, Cistine Chapel and the famous

PietaImplication: He was suspected of poisoning himValue judgment: That’s not right

FACTS: property, manner, set membershipEXPLANATION AND RESULTS: Reason, outcome, causalitySYNTHESIS: conclusions, positions, viewpoints

Total number of statements by type recorded at each stage of the data collection process

Statement Type WT1 N WT2 N WT3 N

Facts Number of statements

Number of statements

Number of statements

Properties 1194 1820 2160

Manner 589 1190 1514

Set membership 143 218 343

Explanation and result Number of statements

Number of statements

Number of statements

Reason 152 163 193

Outcome 35 58 89

Causality/consequence 48 105 116

Synthesis 29 70 82

PatternsSUBSTANCE: Students represented their topic predominantly by factual property and manner statements, and that they used increasingly more factual statements to represent their knowledge throughout the three stages.

NUMBER OF STATEMENTS: There were significant differences between the number of property statements that the students used between WT1 and WT2 (t (220) =6.29, p = .0000; between WT2 and WT 3 (t(222) =4.26, p <=.00)

Overall the number of reason and outcome statements were lower than property and manner statements, and did not substantially increase as students progressed through the stages.

Students appeared to be oriented to gathering facts and knowing a set of facts throughout the search process.

Two predominant patterns: Additive and Integrative

ADDITIVE APPROACH

Knowledge development characterized by progressive addition of property facts

As the students built knowledge, they continued to add property and manner statements, and to a lesser extent, set membership statements

Stockpile of facts, even though facts were sorted, organized and grouped to some extent into thematic units by conclusion.

Remained on a descriptive level throughout

Horizontal

WT1: He is very famous for his plays …….(100, 285001)

WT2: He married Anne Hathaway. They had 3 children. ……He wrote 37 plays and 152 sonets. …… (100, 285001)

WT3: He was born on April 23, 1564 in Warichshire, Stratford-upon-Avon, Britain………Married at age 18. Had three children; Judith, Hammet and Susana. He was the first boy in the family, had 3 sister and 1 brother, Joan, Margaret, Gilbert sibling.(100, 285001)

INTEGRATIVE APPROACH

Initial: superficial sets of properties

Moved beyond gathering facts:- building explanations- address discrepancies- organizing facts in more coherent ways

Interpret found information to establish personal conclusions and reflect on these.

Some students subsumed sets of facts into fewer but more abstract statements at the end

Immune reactions

WT1: It probably has most to do with how the body reacts to certain problems in the body. Like how a body reacts to a sneeze or a cough.

WT2: The immune system is what protects you, anybody, from various outside bacterias, viruses, and germs. …….The immune has I-cells and other types of cells that help fight the……When you cut yourself you can see the immune system at work because you can see the cells that are rebuilding the tissues that were cut………(long reply)

WT3: (very long reply) The immune system was a big topic. I found out that there is actually two types of immune systems in the body. ………I learned that if a bacteria enter your body, it could enter a cell, replicate in a manner of minutes and if there are…..So you could have millions of bacteria in the body after an hour…….the body works against such organ degration, mineral deficiency, mechanical damage and other. What I really enjoy to learn was that the minute you are born….and when you die your immune system shuts down letting in all the bad stuff, so now the body is an open door….

Structure of Knowledge

Ideas are discrete and unrelated.

Some limited structure evident –meaning more than one instantiation- some ideas are joined or linked (grouped) while others are discrete or unrelated.

Contiguous ideas are associated; set of ideas may be somewhat continuous.

Overall, ideas are interrelated and continuous.

Ideas are integrated and unified; there is structural centrality, and overall unification.

Changes in Structure of Knowledge

Generally there was a change towards more structured replies from WT1 to WT3, but it was not consistent.

Change from disparate, unstructured and random listing of facts to endpoint representations that showed some organization of facts into thematic groupings

For students showing an additive model of knowledge structure, WT1 showed more structure than WT2. This is consistent with the pervasive viewpoint that people’s existing knowledge is structured.

However, as students built up background knowledge, the gathering of facts rather than the sequencing and organization of these facts seemed to take over, as evident in WT2 representations being less structured and interlinked.

By WT3, some organization of these facts into conceptually coherent groups and linked together had taken place, particularly for students showing integrative model

Labeling (Title) of Knowledge

Little investigation done on the analysis of titles given to a representation of knowledge

Given that the library profession is built on classification schemes that reflect knowledge of the domain being classified, we assumed a relationship between title and substance of knowledge

We assumed that as content changed, this would be reflected in a change of title.

Three Kinds of Titles

1. General title: A title that describes the project on a general, overall level. For example: Ancient Rome; pollution; The West

2. Specific title: The title brings forward a specific

aspect of the project. For example: Ancient Roman Government; Rulers of Ancient Rome; air pollution; westward expansion

3. The title is expressed in a creative, or artistic way: For example: The best and most famous poet who ever lived-William Shakespeare; PCP, not just a drug or PCP, deadly, yet not gone; Is your hair really getting clean?

Patterns in Changes of Title(60%) The title remains the same and typically general throughout. There was no change in how the student stated the title, for example: Ancient Greece Ancient Greece Ancient Greece (20%) Hourglass phenomenon, where a title is general in WT1 and WT3, but specific in WT2. For example: Food ancient food food; Health in Ancient Greece medicine in ancient Greece Health in Ancient Greece; Increasing specificity: For example Ancient Greece Fashion in Ancient Greece Daily clothing of Ancient Greece; Julius Caesar the life of Julius Caesar Caesar: his deathGeneral to specific to artistic/creative: For example: Entertainment ancient Roman entertainment what the Romans did for fun; perfume and cologne chemistry behind the scent chemistry stinks, the chemistry behind perfumes and other scentsAssociative changes: (small numbers) Roman military Roman fashion Food in RomeGeneral to general to artistic: (small numbers) The culture of the English people The culture of the English people The mother of America but still so different.

Change in Titles

The title the students give their projects may not accurately reflect their level of knowledge

Use of general titles throughout seems to indicate that they simply replied the question by stating their “official” project title instead of realizing the possibility of expressing their own personal conceptualization of it.

This seems to be an outcome of instructions for the completion of the questionnaires, rather than the students’ own misunderstandings of the question.

Some students whose titles seemed to develop to a more specific expression (or an hourglass expression) over time had a synthesized reply in WT3.

Estimate of Knowledge

We asked the students to estimate how much they thought they knew about their topic as they progressed through the task. A five-point scale was used: nothing (0), not much (1), some (2), quite a bit (3), and a great deal (4).

On average the students own estimated knowledge developed from “not much” (mean = 1.26, SD =.91) at WT1, to “some” or “quite a bit” (mean = 2.6, SD =.83) at WT2 At WT3, most students found they knew at least “quite a bit” (mean = 3.2, SD =.76) about their topics.

There was a significant difference between the students’ estimates of their own knowledge between the stages

Did the students’ knowledge change?

Substance of KnowledgeStructure of KnowledgeAmount of KnowledgeExtent of KnowledgeTitle of Knowledge

Knowledge outcomes of instruction: depth vs widthInstructional interventionsZones of intervention