promoting student engagement, interest, and understanding in...

Promoting Student Engagement, Interest,

and Understanding in Science,

Technology, Engineering, and Math (STEM):

An Evaluation of the GLOBE/LEAD Skyscrapers Enrichment

Programs

Evaluation Report

November 2013

Prepared by: Youth Studies, Inc

155 Water Street

Brooklyn, NY 11201

GLOBE/LEAD Skyscrapers (2012-13) Evaluation Report

Table of Contents

Executive Summary ........................................................................................................................ 3

Background ..................................................................................................................................... 4

Description of Evaluation Process .................................................................................................. 5

Participant Outcome Findings......................................................................................................... 7

Students’ Comprehension of Scale and Proportion..................................................................... 7

Students’ Confidence in their Ability to Succeed in Math and Science ..................................... 9

Students’ Future-oriented Motivation to Pursue Math and Science Careers ............................ 11

Students’ Knowledge of Basic Engineering and Architecture Concepts and Definitions ........ 12

References ..................................................................................................................................... 15

Appendix A: Skyscrapers Participant Assessment ....................................................................... 16


Executive Summary

This report includes findings from YSI’s outcomes evaluation of the Guided Learning

through Our Built Environment (GLOBE) and Learning through Engineering, Architecture, and

Design (LEAD) programs implemented during the 2012-13 school year. These programs are a

collaboration between several New York City public schools and the Salvadori Center, a not-for-

profit educational organization that uses the principles of architecture and engineering to help

students in schools and out-of-school time programs to master mathematics and science

concepts. GLOBE and LEAD aim to improve students’ engagement, motivation, and

understanding of important science, technology, engineering and mathematics (STEM) content

through high-quality, hands-on, project-based learning experiences.

During the 2012-2013 school year, Youth Studies, Inc. administered pre- and post-

assessments to students participating in the 23-week GLOBE program and the 8-week LEAD

program. In both programs, Salvadori educators implemented in school residencies during which

the Salvadori-designed Skyscrapers curriculum was taught. Skyscrapers uses hands-on activities

and design challenges that aim to promote student understanding of import STEM concepts

including forces, simple machines, scale, proportion, and the design process. A total of 381

students participated in the assessment process across both programs.

The following are key highlights from YSI’s evaluation of the Skyscrapers initiative:

• A total of 381 students participated in YSI’s assessment of the Skyscrapers program.

Forty-nine percent of GLOBE participants and 43 percent of LEAD participants were

female. Fifty-three percent of GLOBE participants and 27 percent of LEAD participants

self-identified as “Black” or “Hispanic/Latino.”

• Both GLOBE and LEAD participants demonstrated a statistically significant increase in

their comprehension of scale and proportion concepts relevant to the Skyscrapers

curriculum (See Page 7).

• Both GLOBE and LEAD participants demonstrated a significant increase in their

confidence that they can be successful in math and science as measured by the Fennema-

Sherman Attitudes Scale, a math and science attitude scale that has been used extensively

in education research (See Page 9).


motivation to pursue educational and career choices in math and science as measured by

the assessment items from the Programme for International Student Assessment (PISA)

(See Page 11).


knowledge of core engineering, design, and architecture concepts relevant to the

Skyscrapers curriculum (See Page 12).


Background

The Guided Learning through Our Built Environment (GLOBE) and Learning through

Engineering, Architecture, and Design (LEAD) programs are ambitious, comprehensive school

enrichment projects that seek to transform young people’s engagement, interest, and

understanding of science, technology, engineering and mathematics (STEM) subject matter

through high quality, project-based learning experiences. These program highlight important

engineering and design concepts through hands-on investigations of the built environment with

an emphasis on collaborative learning. GLOBE is an in-school residency during which Salvadori

Center educators implement one of several available Common Core-aligned curricula over the

course of a full school year (23 weeks). The Salvadori-designed curriculum is presented to

students through weekly 45-minute classroom lessons. The Salvadori Center also provides a

condensed, 8-week version of this in-school residency model called Learning through

Engineering, Architecture, and Design (LEAD).

During the 2012-13 school year, Youth Studies, Inc. (YSI) conducted an outcomes

evaluation of the GLOBE and LEAD programs. In both programs, the Salvadori Center

educators implemented the Skyscrapers curriculum, which uses hands-on activities and design

challenges to foster student learning and exploration of topics relating to skyscrapers. These

topics include forces, simple machines, scale, proportion, and design. Skyscrapers uses the

principles of architecture and engineering to help students master mathematics and science

concepts. The curriculum consists of a series of hands-on sessions focused on important

background knowledge and several sessions dedicated to a final project: Modular Skyscrapers.

The Skyscrapers curriculum is aligned to the Common Core Math Standards and the New York

State Standards for Math, Science, and Technology and emphasizes important science and math

skills including measurement, data collection, making predictions, drawing conclusions, use of

scale and proportion, application of the design process, and presentation of findings.

Skyscrapers activities are designed to reinforce the scientific inquiry process of

investigating, hypothesizing, testing, and drawing conclusions. Below is an overview of the

topics covered in the Skyscrapers curriculum:

• An Introduction to Skyscrapers: Students learn about the historical context of

skyscrapers and identify the characteristics that make skyscrapers unique.

• Skyscrapers and Technology: Students learn about the technological advances

that made skyscrapers possible, and create a fixed pulley system.

• Structural Supports for Skyscrapers: Students learn how buildings are

supported by a structural grid, and how a column’s shape affects its load-bearing

capacity.

• Types of Beams: Students learn how the height of a column affects its load-

bearing capacity, and how the shape and length of a beam affects its strength.


• Tools of Architects: Measurement, Scale, and Proportion: Students learn how

architects and engineers use measurement, scale and proportion in building

design.

• Structural Grids: Students learn how scale is used to make architectural

drawings, and how columns and beams come together to make a structural grid.

• The Design of Skyscrapers: Students learn how the design of a curtain wall is a

reflection of the architect’s style and the building’s function.

• Introduction to the Design Challenge: Students learn how pre-fabricated

shipping containers can be used to build modular skyscrapers.

• Modular Skyscrapers: The Design Process: Students are exposed to the design

process and learn about brainstorming, sketching, presenting their ideas to others,

and incorporating the feedback of others into their final plan.

• Building a Modular Skyscraper: Students construct a modular skyscraper.

• Final Presentations: Students conduct a collaborative oral presentation.

The rest of this report summarizes current findings from YSI’s evaluation of the GLOBE

and LEAD Skyscrapers program.

Description of Evaluation Process

Participant Assessments

YSI administered pre- and post-participation student assessments at participating GLOBE

and LEAD programs. In addition to basic background questions (e.g. gender, age, and ethnicity),

the pre- and post-assessments included standardized measures of: 1) an assessment of students’

comprehension of scale and proportion concepts; 2) students’ confidence in their ability to

succeed in math and science; 3) students’ future-oriented motivation to pursue math and science

careers and; and 4) students’ knowledge of architectural and engineering concepts relevant to

skyscrapers.

A total of 381 students participated in YSI’s assessment of the Skyscrapers program. Of

those 381 students, 294 participated in the GLOBE program and 87 participated in LEAD.

Moreover, a total of 282 students completed both a pre- and post-test assessment (including 221

GLOBE students and 61 LEAD students). The remaining 99 students participated in the pre-test

only (47) or only submitted a post-test assessment (52). Table 1 includes some background

characteristics for the 381 students who participated in the evaluation. As seen in Table 1, 49

percent of GLOBE participants and 43 percent of LEAD participants were female. Fifty-three

percent of surveyed GLOBE students self-identified as “Black” or “Hispanic/Latino.” By

comparison, 27 percent of LEAD students self-classified as “Black” or “Hispanic/Latino.”


Table 1. Background Characteristics of GLOBE and LEAD Participants

Background Characteristic GLOBE

(%)

LEAD

(%)

Grade

1st - -

2nd

- -

3rd

22 100

4th

11 -

5th

9 -

6th

58 -

7th - -

8th

- -

Gender

Female 49 43

Male 51 57

Ethnicity *(categories are not mutually exclusive)

Black or African-American 5 16

Hispanic/Latino (of any race) 48 11

White 30 62

Native American 2 10

Asian 19 10

Other 10 22

To assess how Skyscrapers participants’ attitudes about math and science may have

changed over the course of the program, YSI evaluators included survey items from the

Fennema-Sherman Attitudes Scale, a math and science attitude scale that has been used

extensively in education research. Using students’ responses to questions from the Fennema-

Sherman Attitudes scale, we constructed measures of students’ personal confidence in their

ability to do math and science. These attitudes were assessed prior to and after students

participated in the GLOBE and LEAD Skyscrapers program. More specifically, students were

asked in both pre- and post-test surveys to agree or disagree with the following statements

related to these attitudes. Students’ responses to similar statements were averaged to form

measures of students’ confidence in math and science.

Table 2. Modified Fennema-Sherman Attitude Scales

Confidence Items

Math is hard for me.

Science is hard for me.

I know I can do well in math.

I know I can do well in science.

I am sure I can learn math.

I am sure I can learn science.

I think I could do advanced math and science.


Moreover, three survey items from the Programme for International Student Assessment

(PISA) were included to assess students’ future-oriented motivation to pursue science education

and careers (OECD, 2007). Those items are listed in Table 3.

Table 3. PISA Future-Oriented Science Motivation Scale

Future-Orientated Science Motivation

I would like to work in a career involving science.

I would like to study science when I go to college.

I would like to work on science projects as an adult.

To assess changes in participants’ knowledge of specific concepts related to the

Skyscrapers curriculum, participant surveys included a set of 12 “content” items (8 in the LEAD

assessment) that were drawn directly from the Salvadori curriculum. A copy of the GLOBE

survey instrument is included as Appendix A1.

Participant Outcome Findings

Skyscrapers participants were surveyed at the beginning and conclusion of the 23-week

GLOBE program cycle and 8-week LEAD program during the 2012-13 school year. In addition

to basic background questions (e.g. gender, age, and ethnicity), the pre- and post-

assessments included standardized measures of: 1) students’ comprehension of scale and

proportion; 2) students’ confidence in their ability to succeed in math and science; 3)

students’ future-oriented motivation to pursue math and science careers; and 4) students’

knowledge of architectural and engineering concepts relevant to skyscrapers.

Students’ Comprehension of Scale and Proportion

The GLOBE pre- and post-test questionnaires included three performance tasks that were

designed to assess students’ understanding of scale and proportion concepts central to the

Skyscrapers curriculum. The LEAD assessment included two of these items as indicated in Table

4 below.

1 The LEAD survey assessment is a slightly condensed version of the GLOBE questionnaire.


Table 4. Scale and Proportion Assessment Items

Question # Assessment Item

21 (Included in

GLOBE

Assessment Only)

The picture below shows Duquan’s scale drawing of his classroom. Which scale did he most likely use?

22

Please estimate the height of the building based on the scale drawing above: ________ meters

23

In the graph below, please sketch the building to a larger scale where every square is equal to 50 meters.

YSI created an overall measure of student comprehension that summarizes how well

students performed on these tasks. Possible values for this measure ranged from 0 (indicating 0

correct responses) to 100 (indicating that the students performed all of the tasks correctly).

As seen in Table 5 below, GLOBE and LEAD participants demonstrated a significant

increase in their comprehension of scale and proportion concepts. A paired-samples t-test was

conducted to compare students’ knowledge of scale and proportion at the beginning of the

Salvadori program and after the Skyscrapers module was completed. There was a significant

improvement in pre-test vs. post-test conditions among GLOBE students (Mdifference=12.3,

SD=33.8); t (191)=5.05, p < .000. YSI also observed a significant improvement in Scale and


Proportion comprehension among LEAD students (Mdifference=20.2, SD=41.4); t (41)=3.17, p <

.003). Figure 1 below presents a visual representation of the pre-test vs. post-test comparisons.

Table 5. Pre- vs. Post-test Assessments of Students’ Comprehension of Scale and Proportion

Student Understanding of

Scale and Proportion

GLOBE Program LEAD Program

Mean (range 0-100)

Pre-test Score 62.7 40.5

Post-test Score 75.0 60.7

Change + 12.3# + 20.2

#

# Statistically significant change from baseline to follow-up (p<.00)

Figure 1. Pre- vs. Post-test Assessments of Students’ Comprehension of Scale and Proportion

Note: The circle and diamond markers indicate the pretest and posttest score mean performance on this measure and

the brackets above and below these mean scores display 95 percent confidence intervals for these mean scores.

Students’ Confidence in their Ability to Succeed in Math and Science

To assess how GLOBE and LEAD students’ attitudes about math and science may have

changed during the year they participated in the program, evaluators administered a modified

version of the Fennema-Sherman Attitudes Scale (see description above). Responses to this


assessment were used to develop a measure of students’ personal confidence in their ability to do

math and science. These attitudes were assessed prior to and after students participated in

Skyscrapers. The assessment items included:

Table 6. Student Confidence Items

Question # Confidence Item

(Answer Choices: Strongly Disagree, Disagree, Agree, Strongly Agree)

6 I am sure I can learn math

7 I know I can do well in science

8 I think I could do advanced math and science

9 Math is hard for me

10 I know I can do well in math

13 Science is hard for me

15 I am sure I can learn science

As seen in Table 7 below, both GLOBE and LEAD participants demonstrated a

significant increase in their confidence that they can be successful in math and science. A paired-

samples t-test was conducted to compare students’ self-efficacy at the beginning and completion

of the Skyscrapers program. There was a significant improvement in pre-test vs. post-test

conditions among GLOBE students (Mdifference=0.54, SD=0.35); t (196)=21.51 p = .000). YSI

also observed a significant improvement in math and science efficacy among LEAD students

(Mdifference=0.58, SD=0.34); t (35)=10.22, p < .000). Figure 2 below presents a visual

representation of the pre-test vs. post-test comparisons.

Table 7. Pre- vs. Post-test Assessments of Students’ Confidence in their Ability to Succeed in

Math and Science

Math and Science Confidence


Mean (range 1-4)



Change + 0.54# + 0.58

#



Figure 2. Pre- vs. Post-test Assessments of Students’ Confidence in their Ability to Succeed in

Math and Science

Students’ Future-oriented Motivation to Pursue Math and Science Careers

The GLOBE and LEAD participant surveys included three items to measure students’

motivation to pursue future education and careers in math and science. These items included:

Table 8. Future-Oriented Motivation Assessment Items

Question # Motivation Item

(Answer Choices: Strongly Disagree, Disagree, Agree, Strongly Agree)

11 I would like to work in a career involving science.

12 I would like to study science when I go to college.

14 I would like to work on science projects as an adult.

As seen in Table 9 below, GLOBE and LEAD participants demonstrated a significant

increase in motivation to pursue educational and career choices in math and science. A paired-

samples t-test was conducted to compare students’ future-oriented motivation to pursue math and

science as measured at the beginning and completion of the Skyscrapers program. There was a

significant improvement in pre-test vs. post-test conditions among GLOBE students

(Mdifference=0.39, SD=0.76; t (195)=7.22 p = .000). YSI also observed a significant improvement

in motivation among LEAD students (Mdifference=0.53, SD=0.68; t (46)=5.30, p < .000). Figure 3

below presents a visual representation of the pre-test vs. post-test comparisons.


Table 9. Pre- vs. Post-test Assessments of Students’ Future-Oriented Science Motivation

Math and Science Motivation


Mean (range 1-4)



Change + 0.39# + 0.53

#


Figure 3. Pre- vs. Post-test Assessments of Students’ Future-Oriented Science Motivation

Students’ Knowledge of Basic Engineering and Architecture Concepts and Definitions

The GLOBE and LEAD participant survey included items measuring students’

knowledge of basic engineering, design, and architecture definitions and concepts. These items

were scored and a “Knowledge of Engineering and Architecture” scale was created to measure

the proportion of those questions that a student answered correctly. Possible values for this

measure ranged from 0 (indicating 0 correct responses) to 100 (indicating that the students

answered all questions correctly). These items included:


Table 10. Knowledge of Key Engineering, Design, and Architecture Concepts Assessment Items

Content Knowledge Items

25 A new skyscraper has just been built in downtown Manhattan. What does this change? (Correct Response [CR] = the skyline of Manhattan)

26 Which of the following types of beams is most often used in skyscrapers? (CR = I-beam)

27 The skeleton of a skyscraper is called the … (CR = structural grid)

28 Which of the following two terms are opposite forces? (CR = tension and compression)

29 A characteristic of a curtain wall is that it is … (CR = non-load bearing)

30 (Included in GLOBE

Assessment Only) An exceptionally tall skyscraper with a steel frame is called a … (CR = megastructure)


Assessment Only) Elevators use what type of simple machine? (CR = pulley)

32 Which of the following shapes would make the strongest column? (CR = circle)

33 The use of shipping containers to build new skyscrapers is an example of what type of construction? (CR = modular)

34

Which of the following is a true statement? (CR = The hand and the house are not in proportion to one another)


Assessment Only)

Which beam – a longer beam or a shorter beam – is stronger? (CR = A shorter beam is stronger and will hold a greater load)


Assessment Only)

Which of the following pulley systems requires the least amount of force to lift a 100 lb. weight? (CR = A moveable pulley requires less force to lift a 100 lb weight)

As seen in Table 11 below, both GLOBE and LEAD participants demonstrated a

significant increase in their knowledge of core engineering, design, and architecture concepts

relevant to the Skyscrapers curriculum. A paired-samples t-test was conducted to compare

students’ comprehension at the beginning and completion of the Skyscrapers program. There

was a significant improvement in pre-test vs. post-test conditions among GLOBE students


(Mdifference=25.9, SD=19.0; t (215)=20.05 p = .000). YSI also observed a significant improvement

in knowledge gain among LEAD students (Mdifference=29.4, SD=26.2; t (58)=8.62, p < .000).

Figure 4 below presents a visual representation of the pre-test vs. post-test comparisons.

Table 11. Pre- vs. Post-test Assessments of Students’ Knowledge of Engineering, Design, and

Architecture Concepts

Student Comprehension Score


Mean (range 0-100)



Change + 25.9# + 29.4

#


Figure 4. Pre- vs. Post-test Assessments of Students’ Knowledge of Engineering, Design, and

Architecture Concepts


References

Basu, S. J., & Barton, A. C. (2007). Developing a sustained interest in science among urban

minority youth. Journal of Research in Science Teaching, 44 (3), 466-489.

Bouillion, L. M., & Gomez, L. M. (2001). Connecting school and community with science

learning: Real world problems and school-community partnerships as contextual scaffolds.

Journal of Research in Science Teaching, 38 (8), 878-898.

Hamre, B.K., Pianta, R.C., Mashburn, A.J., & Downer, J.T. (2007). Building a science of

classrooms: Application of the CLASS framework in over 4,000 US early childhood and

elementary classrooms. Charlottesville, VA: University of Virginia, Center for Advanced

Study of Teaching and Learning.

Marx, R. W., Blumenfeld, P. C., Krajcik, J. S., Fishman, B., Soloway, E., Geier, R., et al. (2004).

Inquiry-based science in the middle grades: Assessment of learning in urban systemic

reform. Journal of Research in Science Teaching, 41 (10), 1063-1080.

Organization for Economic Cooperation and Development (OECD). (2001). Knowledge and

Skills for Life: First Results from the OECD Programme for International Student

Assessment (PISA), 2000. Paris: OECD.

Pechman, E.M., Mielke, M.B., Russell, C.A., White, R.N., & Cooc, N. (2008). Out-of-School

Time (OST) Observation Instrument: Report of the validation study. Washington, DC: Policy

Studies Associates.

Schmieder, A. A., & Michael-Dyer, G. (1991). State of the scene of science education in the

nation. Public Health Service National Conference.

U.S. Department of Education. (1991). America 2000: An education strategy (Tech. Rep.).

Washington, DC: U.S. Department of Education.

Appendix A: Skyscrapers Participant Assessment

19

Salvadori Center Participant Survey, 2012-13 (Skyscrapers-Baseline) 3

5"

4"

3"

2"

1"

Scale: = 100 meters

The next two questions have to do with the building in the picture above.

22. Please estimate the height of the building based on the scale drawing above.

Write your estimate here: _________ meters

23. In the graph below, please sketch the building to a larger scale where every square is equal to 50

meters (Scale: = 50 meters).

10"

9"

8"

7"

6"

"

5"

4"

3"

2"

1"

Scale: = 50 meters

Note: Your building

doesn’t need to look

exactly like the drawing.

But do your best to make

the size correct.

22

Salvadori Center Participant Survey, 2012-13 (Skyscrapers-Baseline) 6

34. Which of the following is a true statement?

�1 The hand and the house are not in proportion to one another.

�2 There is no such thing as a house that is painted yellow.

�3 The house is real and the hand is fake.

�4 The house is an example of a modular building.

�5 I don’t know.

35. Which beam – a longer beam or a shorter beam – is stronger?

�1 A shorter beam is stronger and will hold a greater load.

�2 A longer beam is stronger and will hold a greater load.

�3 Both beams are equally strong.


36. Which of the following pulley systems requires the least amount of force to lift a 100 lb. weight?

A) Fixed Pulley System: B) Moveable Pulley System:

�1 A fixed pulley (A) requires less force to lift a 100 lb. weight.

�2 A moveable pulley (B) requires less force to lift a 100 lb. weight.

�3 Both pulleys require an equal amount of force to lift a 100 lb. weight.


THANK YOU!

promoting student engagement, interest, and understanding in...

Documents