measuring the impact of stem learning in afterschool · 2016. 11. 2. · of stem afterschool +...
TRANSCRIPT
![Page 1: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/1.jpg)
Measuring the impact of STEM learning in afterschool
The webinar will begin shortly.
![Page 2: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/2.jpg)
Audio difficulties? Keep this number handy!Dial: 1-877-860-3058Code: 1135574
Experiencing Delays?Try closing out the other programs running on your computer.
Have a question or comment?Use the group chat to interact with presenters and other participants.
Housekeeping Notes
![Page 3: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/3.jpg)
Today’s Speakers
Kevin CrowleyProfessor of Learning Sciences
& Policy, Univ. of Pittsburgh
Bronwyn BevanSenior Research ScientistUniversity of Washington
Robert TaiAssociate Professor of Education, Univ. of Virginia
Vera MichalchikDirector of Evaluation and
Research, Stanford University
![Page 4: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/4.jpg)
Webinar Overview
1. Introduction (Bronwyn)
2. Activated learning (Kevin)
3. Connected learning (Vera)
4. Longitudinal views (Robert)
5. Panel Questions
6. Audience Q&A
![Page 5: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/5.jpg)
New Strategies for Documenting
Learning in Afterschool: An
Ecological ApproachBronwyn Bevan, University of Washington
![Page 6: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/6.jpg)
+ A process that happens over time and across setting
Learning: An Ecological Approach
![Page 7: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/7.jpg)
+ A process that happens over time and across setting
+ A process that involves identity development
+ “I want to do this”
+ “I can do this”
+ “This matters to me, my future, my community”
+ A process mediated by cultural resources
+ Language
+ Norms for social interaction
(e.g, group versus individual; verbal versus non-verbal,
etc.)
+ Societal values and labels that communicate what
is important; who is good at what, etc.
Learning: An Ecological Approach
![Page 8: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/8.jpg)
+ MOST COMMON: Learning is an outcome
+ Interest, attitudes, and motivations to do STEM
+ STEM career awareness
+ Surveys measuring short term pre/post changes
+ NEW APPROACHES: Learning is a process
+ Documenting over time
+ Making connections across settings
+ Situates the afterschool experience as an important
contributor to longer-term processes
Measuring Impacts
of STEM Afterschool
![Page 9: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/9.jpg)
+ ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh)
+ CONNECTED LEARNING (Vera Michalchik, Stanford U)
+ LONGITUDINAL VIEWS (Robert Tai, U Virginia)
NEW ***Measuring Impacts*** NEW
of STEM Afterschool
![Page 10: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/10.jpg)
+ Develop R+P Tools and Tools for R+P
+ Create R+P Conversations and Exchanges
+ Build and Study Research-Practice
Partnerships (RPPs)
researchandpractice.org
Research+Practice Collaboratory
![Page 11: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/11.jpg)
![Page 12: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/12.jpg)
Science Learning Activation
What positions youth
for success in science/STEM?
How can we activate children’s interest and
curious minds in ways that ignite persistent
engagement in science learning and inquiry?
![Page 13: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/13.jpg)
Tracking what changes…
within and across STEM
learning settings
![Page 14: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/14.jpg)
Tracking what predicts…
Scientist/ Engineer
Other STEM Career
Other Career
STEM Literacycritical thinking, evidence based decision makers, engaged citizens
Success
Science (STEM) learning activation =
A composition of dispositions, skills, and knowledgethat enables success in proximal science (STEM) learning experiences.
Activation
![Page 15: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/15.jpg)
What is Activation? The positive feedback loop between activation and success
SuccessActivation
Choice
Perceived Success
Values
Fascination
Scientific Sensemaking
Competency Beliefs
Learning
Engagement
![Page 16: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/16.jpg)
Science Activation Dimensions
o Fascination with natural and physical phenomena.A person’s emotional and cognitive attachment with science topics and tasks.
o Values science. The degree to which a person values science, including the knowledge learned in science, the ways of reasoning used in science, and the role that science plays in families and communities.
o Competency Beliefs. The extent to which a person believes that s/he is good at science.
o Scientific Sensemaking. The degree to which a person engages with science learning as a sensemaking activity. Sub-dimensions include: questions, experiment, evidence, explanation, and nature of science.
![Page 17: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/17.jpg)
Success Dimensions
o Choice. Choosing to participate in the next science learning opportunity (e.g. camp, museum visit, watching a science program).
o Engagement. Includes affective, behavioral, and cognitive components (e.g. excited about materials, doing the science activities at hand, and thinking about science ideas).
o Perceived Success. Feeling successful in completing science learning tasks in absolute and relative terms.
o Learning. Achieving the learning goals for a particular science experience.
![Page 18: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/18.jpg)
What the research says…
o Activation is real. Literature reviews, life-history interviews with people who’ve succeeded in science, and deep qualitative work with youth.
o Activation is measurable. Multiple-choice surveys administered on iPads or paper, customizable to program, strong psychometrics, rigorous validity testing.
o Activation predicts success. Longitudinal studies suggest that the feedback loop works. It doesn’t just work a single way.
o Activation is useful. Design partnerships in Pittsburgh, California and beyond provide common language, measurable outcomes, and big picture thinking.
![Page 19: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/19.jpg)
Activation is available…
![Page 20: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/20.jpg)
Capturing
Connected Learning
When and Where It Happens
Vera Michalchik
(on behalf of CLRN survey team—Bill Penuel, lead)
![Page 21: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/21.jpg)
![Page 22: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/22.jpg)
CONNECTED
LEARNING
PRINCIPLES
Interest
Powered
Centered on youths’ interests—
development of related knowledge and skill
Peer
Supported
Encouraged by peers—
who work together and give feedback
Production
Centered
Making, production, or performance—
for a real audience
Shared
Purpose
Adults participate alongside youth—
youth have a say in the goals and structure
Openly
Networked
Well-resourced—
tools and guidance in using tools
Pursuit is experiences as...
![Page 23: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/23.jpg)
The Instruments http://researchtools.dmlhub.net/
![Page 24: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/24.jpg)
CONNECTED
LEARNING
PRINCIPLES
Interest
Powered
Centered on youths’ interests—
development of related knowledge and skill
Peer
Supported
Encouraged by peers—
who work together and give feedback
Production
Centered
Making, production, or performance—
for a real audience
Shared
Purpose
Adults participate alongside youth—
youth have a say in the goals and structure
Openly
Networked
Well-resourced—
tools and guidance in using tools
Pursuit is experiences as...
![Page 25: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/25.jpg)
![Page 26: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/26.jpg)
Targeted at youths’ experiences in relation to programs
It feels like family when I come here.
I belong here.
I can take risks when I am at this program.
PROGRAM
EXPERIENCES
![Page 27: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/27.jpg)
Civic Engagement
Connections to others
Support for building connections (brokering)
Career orientation
Personal fulfillment and satisfaction
CONNECTED
LEARNING
OUTCOMES
![Page 28: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/28.jpg)
Interview designed to elicit youth’s experiences of
how interest-related activities develop and prepare
youth for their imagined futures.
EXPERIENCES
OF INTEREST-
RELATED
PURSUITS
![Page 29: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/29.jpg)
THANK YOU!!
CLRN resources for program evaluations include:
A website with instruments:
http://researchtools.dmlhub.net/
A self-paced online course:
http://dmlcommons.net/2016-course/
![Page 30: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/30.jpg)
Measuring the Impact of STEM Learning in Afterschool:
A Longitudinal View
ROBERT H. TAI, ED. D.UNIVERSITY OF VIRGINIA
Afterschool Webinar
Afterschool Alliance
November 2, 2016
![Page 31: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/31.jpg)
Active Learning versus
Receptive Learning
Active
Learning
Discovering
Competing
Collaborating
Performing
Caretaking
Teaching
Creating/MakingReceptive
LearningWatching
Reading
Listening
Practicing
Reciting
![Page 32: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/32.jpg)
Framework for Observing and Categorizing Instructional Strategies (FOCIS)
Active
Learning
Discovering
Competing Collaborating
Performing Caretaking
Teaching
Creating/Making
Afterschool STEM
Targets Active Learning
![Page 33: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/33.jpg)
Active Learning Preference Survey
To calculate the Collaboration preference score,
the question responses are averaged.
To calculate the Competition preferences score,
the question responses are averaged.
Etc. for each of the other five active learning types
![Page 34: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/34.jpg)
Example of a Pre- and Post-Program Outcome Comparison, n=39
Collaboratingand
CompetingPreferences
grew stronger
Create/Make, Discover,
Caretake, & Teach
Preferences were strong
to begin with and remained
strong
PerformPreferences
were not affected
All Learning Activity Preferences had positive ratings both before and after the program.
![Page 35: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/35.jpg)
FOCIS Program Evaluation Instrument
• Currently being used by the Boy Scouts of America in the development of their new STEM Scouts Program.
• Evaluation Program planned for DonorsChoose.Org supported by the Overdeck Foundation.
• FOCIS has been used as a longitudinal instrument to track changes in students learning activity preferences in a two-year study. (n=8000+)
![Page 36: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/36.jpg)
We gratefully acknowledge the support of these organizations
All views expressed are those of the researchers and do not represent the views of the National Science Foundation, the Robert N. Noyce
Foundation, or the S. D. Bechtel, Jr. Foundation
![Page 37: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/37.jpg)
The work presented here was based on the efforts of Xaioqing Kong, John T. Almarode, Katherine P. Dabney,
Devasmita Chakraverty, Nathan Dolenc, Angela Skeeles-Worley, and Ji Hoon Ryoo.
I gratefully acknowledge their contributions
Thank you
Robert H. Tai, [email protected]
![Page 38: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/38.jpg)
Panel Questions
![Page 39: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/39.jpg)
Audience Q & A
![Page 40: Measuring the impact of STEM learning in afterschool · 2016. 11. 2. · of STEM Afterschool + ACTIVATED LEARNING (Kevin Crowley, U Pittsburgh) + CONNECTED LEARNING (Vera Michalchik,](https://reader036.vdocuments.us/reader036/viewer/2022071105/5fdf2a9c09d46b227f0f36ce/html5/thumbnails/40.jpg)
Thank you for attending!
@afterschool4all/afterschoolalliancedcAfterschool Snack Blog
Robert [email protected]
Vera [email protected]
Bronwyn [email protected]
Kevin [email protected]
Anita [email protected]