Slice and DiceUsing existing data to answer novel questions about student outcomes
February 7, 2012Jill Kroll
Office of Career and Technical EducationMichigan Department Of Education
Carol ClarkCTE TRAC Director & Program Coordinator
GASC Technology Center
• Recognize challenges in using existing data to answer questions
• Gain knowledge of resources and tools to meet challenges of information needs
2
Learning Objectives
• Two examples - Two solutions–Example 1: Using existing data–Example 2: Overcoming limitations of
existing data
• Resources• Discussion and Questions
3
Overview
Example 1: Using Existing Data
Do secondary CTE students in articulated programs have higher
placement rates?
4
Example 1: Using Existing Data
• What data are available to answer this question?
• In light of the available data, how might you refine the question?
5
Example 1: Using Existing Data
Refine your question - “Placement”–Total placement - “In Employment or
Continuing Education or Military”–Placement in Continuing Education–Placement in Community College–“Related” Placement
•Can you think of other available data?
6
Example 1: Using Existing Data Step 1: Download Follow Up Survey Data
• Log in to CTEIS.com• Follow Up tabReport• Export Building Survey Data
7
8
Example 1: Using Existing DataSave or Open in MSExcel
9
Example 1: Using Existing DataStep 2: CTEIS Ad Hoc Query Tool
• Log In to CTEIS• Choose Reports tab,
Funding Reports, Ad hoc Query• Choose agency, building• 4301 (select year)• Selection criteria
10
Example 1: Using Existing Data Student-Level Data From CTEIS
11
4301-2009-2010
Example 1: Using Existing Data Export Student Data to MSExcel
12
4301-2009-2010
Example 1: Using Existing DataAnalyzing the Data
• Data analysis software–Relational database software–Spreadsheets–Statistical software–Others
13
Example 1: Using Existing Data
Tech Prep Program?2011
Total Placement(Employment or Continuing Education or Military)
Not Placed Yes, Placed Total
Not Tech Prep 562 (5.9%) 9,029 (94.1%) 9,591
Yes, Tech Prep 290 (5.3%) 5,226 (94.7%) 5,516
Total 852 14,255 15,107
14
Example 1: Using Existing Data
15
Tech Prep Program?
2011Attending School
Not Attending School
Attending School Total
Not Tech Prep 2,244 (23.4%) 7,347 (76.6%) 9,591
Yes, Tech Prep 1,212 (22.0%) 4,304 (78.0%) 5,516
Total 3,456 11,651 15,107
Example 1: Using Existing Data
16
Tech Prep Program?
2011Attending A Community College
Attending Another School Type*
Attending a Community College
Total
Not Tech Prep 3,898 (53.6%) 3,381 (46.4%) 7,279
Yes, Tech Prep 2,274 (53.1%) 2,010 (46.9%) 4,284
Total 6,172 5,391 11,563
*Attending business or trade school, college or university, military training or other training
Example 1: Using Existing Data
17
Tech Prep Program?
2011Related Placement
in Continuing EducationUnrelated Placement Related Placement Total
Not Tech Prep 1,249 (24.4%) 3,866 (75.6%) 5,115
Yes, Tech Prep 653 (21.7%) 2,359 (78.3%) 3,012
Total 1,902 6,225 8,127
Example 2: Overcoming Limitations of Existing Data
What is the impact of integrated instruction on academic
achievement?
18
Example 2: Overcoming Limitations of Existing Data
• Local data:–Number of students signed up for each
course–Number of students who took test for
academic credit–-Number of students who passed test–Number of students who signed up for
RAC
19
REQUIRED ACADEMIC CREDIT (RAC)
ELA-12 14
MATH 220
SCIENCE 76
Visual/Performing Arts (VPAA)
65
TOTAL 375
20
GASC 2010-11Signed up for RAC
ELA-12 12
MATH 213
SCIENCE 65
Visual/Performing Arts (VPAA)
62
TOTAL 352
GASC 2010-11 Total Students Achieving RAC
Example 2: Overcoming Limitations of Existing Data
Count % Increase in 2011-12(compared to 2010-11)
ELA-12 15 7%MATH 412 87%SCIENCE 94 24%VPAA 114 75%TOTAL 635
21
GASC 2011-12 Signed up for RAC
Example 2: Overcoming Limitations of Existing Data
CTE Subject Academic Subject
Type of Academic Integration
Method of Qualifying for Academic Credit
Law & Public Safety EnglishLanguage Arts
Instruction by Academic Teacher
Papers graded by English TeacherCTE-specific content, ELA tasks
Multiple Programs(2nd Year Students)
Math Collaborative Teaching Model 1
Pre-Test/Post-Test(Standard Math Test Questions)
Health Science Collaborative Teaching Model
(different for each Health program)
Integrated post-test: Contextual academic test questions specific
to the program
Multiple Programs Visual/Performing Arts
Collaborative Teaching Model
2
Teachers choose a minimum of one project per semester and
grade with a rubric
22
Example 2: Overcoming Limitations of Existing Data
23
Example 2: Overcoming Limitations of Existing Data
Solutions to limitations in existing data:–Modify/Simplify question
24
Example 2: Overcoming Limitations of Existing Data
• Collect supplemental data
25
CTE Subject Academic Subject
Type of Academic Integration
Method of Qualifying for Academic Credit
Law & Public Safety EnglishLanguage Arts
Instruction by Academic Teacher
Papers graded by English TeacherCTE-specific content, ELA tasks
(blind)
Law & Public Safety None None Papers graded by English TeacherCTE-specific content, ELA tasks
(blind)
Example 2: Overcoming Limitations of Existing Data
• Solutions to limitations in existing data–Control extraneous factors
• Choose one subject area• Use three methods of integrating
instruction• Use one method of qualifying for academic
credit (assessing achievement)
26
Example 2: Overcoming Limitations of Existing Data
27
CTE Subject
Academic Subject
Type of Academic Integration
Method of Qualifying for Academic Credit
Validation
Therapeutic Services
Math Academic Instructor Provides Instruction
• Pre-Test/Post-Test using Standard Math Test Questions
• Integrated post-test: Contextual Math test questions specific to the program
College Course
Grade from STARR
DataAccuplacer test results
Same Same Collaborative Teaching Model(CTE instructor
Provides Academic Instruction)
• Same • Same
Same Same Pull-Out Academic Instruction (Tutoring)
• Same • Same
Resources
• Look for more existing data–National
• National Center for Education Statistics (NCES) website
• Midwest Regional Education Laboratory (REL)
–Existing local and state data–Peers
28
NCES: Example Content Areas
Regional Education LaboratoriesFunded by: Institute of Education Sciences (ies)
• Purpose: Provide access to high-quality, scientifically valid education research
• Publications: http://ies.ed.gov/ncee/edlabs/projects/
• Ask A REL: A collaborative reference desk service• Midwest REL: American Institutes for Research
http://www.learningpt.org/
31
Midwest REL: Resources/Services
32
Contact:Jill KrollSupervisor
Office of Career and Technical EducationMichigan Department Of Education
(517) [email protected]
Carol ClarkCTE TRAC Director & Health and Human Services Platform Facilitator
Genesee Area Skill Center Technology Center(810) 760-1444 ext. 176
33