collecting and reporting follow-up data or what’s behind table 5?
DESCRIPTION
Collecting and Reporting Follow-up Data Or What’s Behind Table 5?. American Institutes for Research February 2005. Table 5: Outcomes. Earned a High School Diploma or GED Entered post-secondary School Entered Employment Retained Employment. Table 5: Outcomes. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/1.jpg)
Table 5 and Follow-upTable 5 and Follow-up
Collecting and Reporting Follow-up DataOr What’s Behind Table 5?
American Institutes for ResearchFebruary 2005
![Page 2: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/2.jpg)
Table 5 and Follow-upTable 5 and Follow-up2
Table 5: OutcomesTable 5: Outcomes
Earned a High School Diploma or GED Entered post-secondary School Entered Employment Retained Employment
![Page 3: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/3.jpg)
Table 5 and Follow-upTable 5 and Follow-up3
Table 5: OutcomesTable 5: Outcomes States submit six (6) data points for
each of the four outcomes:
11 2 2 3 3 4 4 5 5 66
Core Follow-up Outcome
Measures
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
![Page 4: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/4.jpg)
Table 5 and Follow-upTable 5 and Follow-up4
Table 5: Column (B)Table 5: Column (B)
All participants who had the outcome as a goal and who exited during the program year.
Core Follow-up Outcome
Measures
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
![Page 5: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/5.jpg)
Table 5 and Follow-upTable 5 and Follow-up5
Table 5: Column (C)Table 5: Column (C)
The number of participants who were surveyed to assess whether they had achieved the outcome.
Note: C should be equal to B unless sampling was used, in which case C would be less than B. For data matching C = [BLANK]
Core Follow-up Outcome
Measures
# Participants With Main or Secondary
Goal
# Participants Included in Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
![Page 6: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/6.jpg)
Table 5 and Follow-upTable 5 and Follow-up6
Table 5: Column (D)Table 5: Column (D)
Of those surveyed or data matched, the number who responded or were used for data match.
Note: D should be equal to or less than B and C each.
Core Follow-up Outcome
Measures
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
![Page 7: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/7.jpg)
Table 5 and Follow-upTable 5 and Follow-up7
Table 5: Column (E)Table 5: Column (E)
Percent of respondents or participants successfully matched; column D divided by
column B.
E = D/B * 100Should be > 50%, but likely not 100%
Core Follow-up Outcome
Measures
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
![Page 8: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/8.jpg)
Table 5 and Follow-upTable 5 and Follow-up8
Table 5: Column (F)Table 5: Column (F)
Number of participants achieving outcome. Note: F must be equal to or less than D.
Core Follow-up Outcome
Measures
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
![Page 9: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/9.jpg)
Table 5 and Follow-upTable 5 and Follow-up9
Table 5: Column (G)Table 5: Column (G)
Percent achieving outcome. G= F/D * 100
Core Follow-up Outcome
Measures
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
![Page 10: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/10.jpg)
Table 5 and Follow-upTable 5 and Follow-up10
Table 5: ExampleTable 5: Example
Core Follow-up Outcome
Measures
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
Entered Employment
1,000 1,000 800 80% 200 25%
![Page 11: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/11.jpg)
Table 5 and Follow-upTable 5 and Follow-up11
Real Data – PY 2002–03 Real Data – PY 2002–03 ExamplesExamples
State
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
State 1 1,056 0 972 0% 356 37%
What type of follow-up was
used?
![Page 12: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/12.jpg)
Table 5 and Follow-upTable 5 and Follow-up12
Real Data – PY 2002–03 Real Data – PY 2002–03 ExamplesExamples
State
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
State 1 1,056 0 972 0% 356 37%
Why is this number less
than column B?
![Page 13: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/13.jpg)
Table 5 and Follow-upTable 5 and Follow-up13
Real Data – PY 2002–03 Real Data – PY 2002–03 ExamplesExamples
State
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
State 1 1,056 0 972 0% 356 37%
What data should be
reported here?
![Page 14: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/14.jpg)
Table 5 and Follow-upTable 5 and Follow-up14
Real Data – PY 2002–03 Real Data – PY 2002–03 ExamplesExamples
State
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
State 1 1,056 0 972 0% 356 37%
Is this percent computed correctly?
![Page 15: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/15.jpg)
Table 5 and Follow-upTable 5 and Follow-up15
Exercise QuestionsExercise QuestionsFor states 2–6: What method of follow-upmethod of follow-up did each
state use? Which states, if any, filled in the
table correctlycorrectly? Which did not? Do you see any errorserrors in the table? Do you have confidenceconfidence in these
data? Why or why not?
![Page 16: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/16.jpg)
Table 5 and Follow-upTable 5 and Follow-up16
Exercise ExampleExercise Example
State
# Participants With Main or Secondary
Goal
# Participants Included in
Survey
# Participants Responding to Survey or
Used for Data Matching
Response Rate or % Available for Match
# Participants Achieving Outcome
% Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)
State 1 1,056 0 972 0% 356 37%
State 2 3,325 NOT AVAIL. 0% 1,062 32%
State 3 8,644 0 0 0% 4,067 47%
State 4 31,363 0 31,363 100% 21,779 69%
State 5 9,633 3,853 1,156 30% 543 13%
State 6 224 224 164 73% 96 59%
![Page 17: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/17.jpg)
Table 5 and Follow-upTable 5 and Follow-up17
Top Errors for Table 5 Top Errors for Table 5 Missing Data
Put “0” for # surveyed in Column (C) Put “0” for # responded or matched in
Column (D) Unreasonably high response rates
Many 100% Unreasonably high data matching Many 100% (DOL goal for matching = 90%)
Unreasonable matching of # who had goal to # who responded
Column (D) = Column (B)
![Page 18: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/18.jpg)
Table 5 and Follow-upTable 5 and Follow-up18
Top Errors for Table 5 Top Errors for Table 5 (cont’d.)(cont’d.)
Incorrectly computed percentages: Response rates calculated as D/C instead
of D/B (B and C should be the same unless using random sampling, however)
Number achieving outcome computed as F/B instead of F/D
![Page 19: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/19.jpg)
Table 5 and Follow-upTable 5 and Follow-up19
20
02
–03
2
00
2–0
3
Data
Colle
ctio
n
Data
Colle
ctio
n
Meth
ods
Meth
ods
![Page 20: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/20.jpg)
Table 5 and Follow-upTable 5 and Follow-up20
Data Matching MethodologyData Matching Methodology Produces better quality data:
More student coverage Better data validity Better response rate Matches Title I—better for OMB common measures
28 states use it for employment measures—we need more states!
What are the reasons for not data matching? How can we promote use of this methodology?
![Page 21: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/21.jpg)
Table 5 and Follow-upTable 5 and Follow-up21
What’s Behind the Data? What’s Behind the Data? State responsibility to ensure accuracy
and quality of follow-up measures Requires monitoring of local program
procedures Key steps:
Identify students—goal setting and tracking Uniform data collection Training Reporting
![Page 22: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/22.jpg)
Table 5 and Follow-upTable 5 and Follow-up22
Percent of Students with Percent of Students with GoalsGoals Measure # with
GoalTotal #
Students% with Goal
Entered Employment—Average ALL states 6,904 52,548 10%*
State 1 417 69,284 1%
State 2 2,624 138,184 2%
State 3 1,609 70,893 2%
Entered post-secondary—Average ALL states 2,919 52,548 6%
State 4 159 33,062 0%
State 5 333 31,574 1%
State 6 14,523 565,311 3%
High School Completion—Average ALL states 7,207 52,548 20%**
State 7 7,192 114,008 6%
State 8 2,768 32,492 9%
State 9 33,118 387,710 9%
* 38% Unemployed (Table 6)** 18% Enrolled in ASE
![Page 23: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/23.jpg)
Table 5 and Follow-upTable 5 and Follow-up23
Quality Control: SurveysQuality Control: Surveys Identify students
Set student goals appropriately By exit quarter for employment measures
Data Collection Quarterly collection for employment State survey instrument Sufficient resources (staff, time, funds) Sampling method (if applicable)
![Page 24: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/24.jpg)
Table 5 and Follow-upTable 5 and Follow-up24
Quality Control: Surveys Quality Control: Surveys (cont’d.)(cont’d.)
Training Trained staff on survey procedures Improving response rate
Reporting Quarterly reporting Database to link outcomes to programs
and students
![Page 25: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/25.jpg)
Table 5 and Follow-upTable 5 and Follow-up25
Quality Control: Data Quality Control: Data MatchingMatching Procedure for collecting and validating
Social Security numbers Report all with goal, prior to eliminating
invalid or missing numbers (Table 5, Col. D) Data have exit quarter and employment
records matched after correct exit quarter
Only students with goal matched and/or reported—should not have 100% with outcome
![Page 26: Collecting and Reporting Follow-up Data Or What’s Behind Table 5?](https://reader035.vdocuments.us/reader035/viewer/2022070403/568139c9550346895da17999/html5/thumbnails/26.jpg)
Table 5 and Follow-upTable 5 and Follow-up26
Discussion: Issues and Discussion: Issues and StrategiesStrategies
Improving reporting and collecting Resources Training on Monitoring