Technology and Engineering Literacy Assessment and Item Specifications for the 2014 National Assessment of Educational Progress

Ch. 1 – Overview

Technology and engineering literacy: the capacity to use, understand, and evaluate technology as well as to understand technological principles and strategies needed to develop solutions and achieve goals. For purposes of this framework, it comprises three areas: Technology and Society, Design and Systems, and Information and Communication Technology.

Things to learn about student in order to give valid assessment:

Reasoning while framing their goals; Planning for projects and the implementation of the plan; Skills in using technologies to gather, manage, and analyze data and information

related to project goals; Capabilities to meet unpredictable challenges that arise during actual, ongoing

problem solving and achievement of goals; Lines of argument in deciding how to alter their approaches in the light of new

evidence; Engagement with peers and experts in addressing goals and deciding how to

achieve them; Deliberations and reasoning when evaluating progress, trade-offs, and results.

Ch. 2 – Areas of Technology and Engineering LiteracyThis chapter describes the essential knowledge and capabilities that will be assessed. They created a high-level breakdown of the content to be assessed. We should create something similar for Computational Thinking.

For each sub-topic, the document includes a list of practices and an assessment rubric. Here are what the two look like for Systems Thinking:

Systems Thinking Key Principles that all students should be expected to understand: Technological systems have parts that work together to accomplish a goal. Systems may include subsystems and may interact with other systems. Systems

may also be embedded within larger systems. Dynamic technological systems require energy with more complicated systems

tending to require more energy and to be more vulnerable to error and failure. Technological systems are designed for specific purposes. They incorporate various

processes that transform inputs into outputs. Two important features of technological systems are feedback and control.

Various methods can be used to increase the reliability of technological systems.

Ch. 3 - Practices and Contexts for Technology and Engineering LiteracyThis chapter describes three generalizable practices that represent the kinds of thinking and application that will be expected of students across the three major assessment areas. The chapter also explains the contexts, or situations and types of problems, in which assessment tasks and items will be set.

Classification of types of assessment targets in the three major assessment areas according to the practices for technology and engineering literacy

Ch. 4 - Overview of the Assessment Design and Specifications for Item Design This chapter provides an overview of the major components of the assessment design.

Types of Tasks and Items• Scenario-based assessment sets

– Long scenarios: 25 minutes (10-15 measures of performance)– Short scenarios: 12-15 minutes (5-10 measures of performance)– Outline: intro (context + motivation) + tasks– Tools: word processing, spreadsheet, presentation tools, engineering

design applications, and communication tools– The students can observe and describe the patterns or characteristics of the

outcomes and can interpret the feedback from the system. They can then evaluate the outcomes of the choices they made in manipulating the components of the system or in using particular tools, and, finally, they can form conclusions

• Discrete item sets– 25 minutes (10-15 stand-alone items)

Response Types:• Short constructed response

– Both scenario based and discrete items• Long (extended) constructed response:

– In long scenario assessment sets– Plus additional measures of students’ responses– Both task and response are extended: the information is contained in

several parts of the overall task– Challenges:

• Enough data is provided• Dependencies among the tasks are minimized• Language capabilities (for four graders or English language

learners).• Selected response:

– Both scenario-based and discrete item sets– Multiple choices: the distracters should appear plausible to students but

should not be justifiable as a correct response, and, when feasible, the distracters should also be designed to reflect current understanding about students’ mental models in the content area

– Computer helps to have a variety of selections: e.g. selection of visual tools!

– In complex, real-world scenarios, it might be the case that there is not a “correct” selection, and in such a situation what matters is that the selection is justified adequately.

– Another form: selecting multiple options. Again, in real word, one “best” combination, but other good combinations too à a scoring rubric is necessary

– Another form: “hot spot” à clicking on a spot on a map, a figure, or a diagram

Ways of Measuring Student Responses:

• Direct responses– Both in discrete item and scenario-based sets– Two types:

• Selection from a set of choices• Written analysis of results and also explanations of why a selection

is made in a scenario• pattern-tracking measures

– The interactions that students get involved in.– Examples: how a student manipulates data, how long it takes +

troubleshooting skills– Types:

• Observation of patterns of action• Tracking the manipulations that the student performs in a scenario

– Can be used to assess certain communication or collaboration skills

Assessment Development• Scenario development

– First to create a shell that outlines the major components of the tasks– Then, the shell can be used to guide the development of tasks and items.

• Discrete item development– The kitchen example (Appendix I)

To develop assessments they use the following form:

Here is a sample completed assessment form:

Scenario-Based Item Sets Examples

In this task, the student must use ICT tools to analyze the impacts of technology on human society.

In this example, students are asked a series of questions related to a simulation of a nuclear reactor.

In this simulation, students navigate among a file manager, an e-mail client, a Web browser, a word processor, and a spreadsheet to make a travel brochure for a fictional town, Pepford.

Descrete Item Sets Examples

In this item represents a sample task in which students must use their knowledge about the engineering design process to answer a constructed response item.

Here, students are given the scenario of a population of small birds— chortlers—whose population is declining. The students are asked to use various tools to analyze data in a variety of ways and to determine some possible causes for the population decrease so that they can then present their findings on the impacts on the chortlers.

In this example, the student is given a route that includes two busy streets. The route takes 25 minutes. The student must drag the route to make it take less time. An additional item could be inserted here that asks students to explain the changes they made, testing their understanding of the trade-off between distance and time.

Here is a sample multiple choice:

Here is a sample ‘hot-spot’

Ways of measuring student responses:

Direct response: inputs directly from the student (selected answers, typed-in justifications, etc.)

Pattern-tracking: measures a student’s interaction with the system. So if the question is to figure out where a system is malfunctioning, this measure looks at how the student tests the system to arrive at an answer. This measure looks for patterns in action.

Ch. 5 – Assessment Design and Student DiversityThis chapter covers things such as:

How to ensure the test is fair to all student (include students with disabilities and English language learners)

How to design the user interface for ease of use Content delivery considerations Who is involved in developing an assessment (psychomatricians, content

experts…) The test development process (trying out questions…)

Ch. 6 - Scoring and Reporting