data collection, publishing, and dissemination of results
DESCRIPTION
Data Collection, Publishing, and Dissemination of Results. Eric Grodsky , University of Minnesota Chandra Muller, University of Texas at Austin. Motivation & Overview. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/1.jpg)
Data Collection, Publishing, and Dissemination of Results
Eric Grodsky, University of Minnesota
Chandra Muller, University of Texas at Austin
![Page 2: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/2.jpg)
Motivation & Overview
The impact of the STEP program and your STEP Type 1 project depend on dissemination of knowledge for application to other settings.
Based on our early experiences working with STEP Type 1 projects
Steps to designing a study, collecting data, publishing and disseminating results: an overview and then example
![Page 3: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/3.jpg)
Study Design
A good study design makes it easier to publish and disseminate results
Begin the design phase early, alongside the program design
Consider collecting baseline (and control group) data before beginning program implementation
![Page 4: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/4.jpg)
Control or Comparison Group
What is the impact of your program on production of STEM majors? To answer this you must be able to estimate: What would have happened without the program? And ideally, what the program did to have the effect,
and for whom?
Consider collecting a control group sample while you are planning the program, before it is implemented Target students for comparison who are similar to those
you will target for the program
![Page 5: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/5.jpg)
IRB: Institutional Review Board
Work with the IRB at your university to gain approval for your study
It is likely that your study will be considered “Exempt”
Example from University of Texas:
![Page 6: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/6.jpg)
Data Collection & Analysis
Study design and response rates are crucial
A randomized control trial—where the only difference between your control and treatment groups is due to random assignment—produces the simplest analysis strategy to show program effects
![Page 7: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/7.jpg)
Dissemination of Findings
Begin analyzing your data and publishing early—pay a writer to write up your results if you and your staff lack time
Early phase results might include: Description of challenges faced by control group Early evidence of climate change in college
As project matures (even at the end of second year phase) results might: Compare control and treatment group progress in STEM courses,
attitudes about STEM, other indicators relevant to your project? Estimate effects on longer range goals of project and the
possible mechanisms through which program works
![Page 8: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/8.jpg)
Where do you publish and disseminate your findings?
Publications of field specific education journals (the education arm of professional organizations, have journals and newsletters, e.g., Journal of Engineering Education) and education journals like Journal of Higher Education, Education and Evaluation Policy Analysis
Early findings and news briefs might go into professional newsletters
Work with your office of public affairs to prepare press releases
Consider an end-of-project book
Get ideas from other STEP projects
![Page 9: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/9.jpg)
Designing the study
What do you want to learn? How/why do you believe that the project succeed?
Who will you serve? What is your target population?
How do you recruit sample members and collect data?
![Page 10: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/10.jpg)
Designing the study
What exactly do you want to ask?
What will you do with all of those data once you have them?
![Page 11: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/11.jpg)
What do you want to learn?
Project will implement intervention you and NSF believe will work What are the explicit objectives? What are the interventions? How/why will they work? (theory)
![Page 12: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/12.jpg)
Example: Professor Yu
What do you want to learn?
Study groups
Chemistry degree
?
![Page 13: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/13.jpg)
theory
Study groups
Chemistry degree
Knowledge
Self-confidence
Enjoyment
![Page 14: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/14.jpg)
The target population
Potential chemistry and physics students Latino and African American students1. Recruitment: High school students 2. Tutoring: In selected classes3. Study groups: In selected classes4. Faculty mentor: in selected classes
![Page 15: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/15.jpg)
Who is in the control condition?
Recruitment: Those not recruited? In comparable non-participating schools? Comparable non-participating classes?
Any way you cut it, must include kids in high school
![Page 16: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/16.jpg)
The target population
Latino and African American students1. Recruitment: At selected schools2. Tutoring: In selected classes3. Study groups: In selected classes4. Faculty mentor: in selected classes
![Page 17: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/17.jpg)
Peer tutoring
Target population: Students enrolled in gateway courses
Control group?
![Page 18: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/18.jpg)
Study groups Target population: those enrolling in
gateway courses in chemistry and physics
![Page 19: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/19.jpg)
Study groups
Control: Entrants to same classes prior to project Entrants to gateway courses in biology
and computer science
Random assignment of sections to study groups or not
![Page 20: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/20.jpg)
Designing data collection
Mode of collection
Sampling procedures
Protocol for recruitment
Timing
The IRB
![Page 21: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/21.jpg)
Mode of collection
Survey Interview (phone or in person) Paper and pencil
Mail In class
Online
Focus group
![Page 22: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/22.jpg)
Sampling
Identify target population
Locate/create sampling frame Make sure you have contact data you need
Structure sample Stratification Matching Blocking
![Page 23: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/23.jpg)
Sampling: The Yu study
Target population: Students enrolled in calculus 101, chemistry 101, biology 101 Each course has 2-4 sections and each
section 6-8 lab sections
![Page 24: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/24.jpg)
Sampling: The Yu study
Randomly assign lab sections to have study groups or not starting in spring of 2013 Sample size of 600
200 students in fall of 2012 from all students enrolled in target courses
100 treatment and 200 control students in spring of 2013
![Page 25: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/25.jpg)
Sampling: The Yu study
Threats to valid inference
![Page 26: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/26.jpg)
Recruiting the sample
Sample nonresponse undercuts validity Shoot for at least 70%
Getting and maintaining sample is often the hardest part of the study
Without a good sample the rest of your evaluation efforts are wasted
![Page 27: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/27.jpg)
Recruiting the sample
Incentives for participation There is actual research on this Cash is king, other incentives (Amazon gift
certificate, MP3 downloads, etc.) not very effective
Some evidence for effectiveness of lotteries, but weak
![Page 28: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/28.jpg)
Recruiting the sample
Initial contact There is actual research on this Letter on letterhead in stamped envelope
if possible In Yu’s case, possibly hand-delivered in
lab
![Page 29: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/29.jpg)
Recruiting the sample
Non-response follow-up Email Snail mail Phone
![Page 30: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/30.jpg)
Data collection timing
Consider last date you can collect data and work backwards One week prior to exams
Snail mail incentive letter 2 email follow ups Snail mail follow up One more email After another week, phone calls
![Page 31: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/31.jpg)
Data collection timing
And of course the IRB…
![Page 32: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/32.jpg)
Instrument design
Goes back to what you want to know Mechanisms by which intervention has
effect Variation in effects of intervention and
what might account for that variation
![Page 33: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/33.jpg)
Instrument design
Do not make up your own items if you can avoid it Consider extant instruments
Ask simple questions respondents can answerNo ‘double barrel’ questions
![Page 34: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/34.jpg)
Instrument design
Survey is a ‘conversation with a purpose’ Script the conversation so it flows Use skip patterns to avoid asking
irrelevant questions Keep is as short as possible
![Page 35: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/35.jpg)
theory
Study groups
Chemistry degree
knowledge
Self-confidence
Enjoyment
![Page 36: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/36.jpg)
Self-confidence: Identity
How much do you agree or disagree with the following statements?
You see yourself as a science person
Others see you as a science person
1=Strongly agree
2=Agree
3=Disagree
4=Strongly disagree
![Page 37: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/37.jpg)
Self-confidence: Self-efficacy
You are confident that you can do an excellent job on tests in this course
You are certain you can understand the most difficult material presented in the textbook used in this course
You are certain you can master the skills being taught in this course
![Page 38: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/38.jpg)
Analyzing Data
Depending on design RCT: difference in means Other designs:
Covariate adjustment Difference in means for certain
subgroups
![Page 39: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/39.jpg)
Analyzing Data
Our example: Does program impact self-confidence? Mean change in self-confidence:
Control students in fall 2012 Control students in spring 2013 Treatment students in spring 2013
Covariate adjustment
![Page 40: Data Collection, Publishing, and Dissemination of Results](https://reader036.vdocuments.us/reader036/viewer/2022070401/568135db550346895d9d4e53/html5/thumbnails/40.jpg)
List of journals for possible publication of STEP Type 1 results (list generated by audience)
• Advances in Engineering Education - http://advances.asee.org/• CBE Life Sciences Education - http://www.lifescied.org/• Chemical Educator - http://chemeducator.org/• Community College Journal of Research and Practice - http://www.tandf.co.uk/journals/authors/ucjcauth.asp• CUR Quarterly - http://www.cur.org/publications/quarterlies.html• Journal of Applications and Practices in Engineering Education - http://japee.net/• Journal of Chemical Education - http://pubs.acs.org/journal/jceda8• Journal of College Science Teaching - http://www.nsta.org/college/• Journal of College Student Retention -http://www.cscsr.org/retention_journal.htm• Journal of Research in Science Teaching - http://www.narst.org/publications/jrst.cfm • Journal of STEM Education: Innovations and Research
http://ojs.jstem.org/index.php?journal=JSTEM&page=index• Journal of Women and Minorities in Science and Engineering -
http://www.begellhouse.com/journals/00551c876cc2f027 • Physics Teacher - http://tpt.aapt.org/• Physics Today - http://www.physicstoday.org/• Portland International Conference on Management of Engineering and Technology (PICMET) -
http://www.picmet.org/main/• STEPcentral.net A Community Forum for NSF STEP Projects - http://stepcentral.net/• Student Affairs in Higher Education - http://www.naspa.org/