quality assurance practices associated with producing ...programs.ccsso.org/content/pdfs/examining...
Post on 17-Jul-2020
2 Views
Preview:
TRANSCRIPT
Council of Chief State School Officers Accountability and State Reporting
Robin Taylor (Chair) Associate Secretary
Delaware Department of Education
J. P. Beaudoin Research in Action, Inc.
Pete Goldschmidt
California State University, Northridge/CRESST
Quality Assurance Practices associated with Producing Cohort Graduation Rates
August 2007
Council of Chief State School Officers The Council of Chief State School Officers (CCSSO) is a nonpartisan, nationwide, nonprofit organization of public officials who head departments of elementary and secondary education in the states, the District of Columbia, the Department of Defense Education Activity, and five U.S. extra-state jurisdictions. CCSSO provides leadership, advocacy, and technical assistance on major educational issues. The Council seeks member consensus on major educational issues and expresses their views to civic and professional organizations, federal agencies, Congress, and the public. State Education Indicators The Council is a strong advocate for improving the quality and comparability of assessments and data systems to produce accurate indicators of the progress of our elementary and secondary schools. The CCSSO education indicators project is providing leadership in developing a system of state-by-state indicators of the condition of K-12 education. Indicators activities include collecting and reporting statistical indicators by state, tracking state policy changes, assisting with accountability systems, and conducting analyses of trends in education. The CCSSO reports on state education policies inform education leaders and educators about the current status and trends in policies across the 50 states that define and shape elementary and secondary education in public schools. The report is part of a continuing biennial series produced by the Council’s education indicators project. We report 50-state information on policies regarding teacher and leader preparation and certification, graduation requirements, state content standards, student assessment programs, school time, and student attendance. The work of CCSSO is possible because of the excellent cooperation and coordination by staff in each state department of education as well as by funding from the U.S. Department of Education.
2007 Council of Chief State School Officers
Elizabeth Burmaster (Wisconsin), President Rick Melmer (South Dakota), President-Elect
Gene Wilhoit, Executive Director
Rolf K. Blank, Director of Education Indicators
Council of Chief State School Officers Attn: Publications
One Massachusetts Ave., NW, Suite 700 Washington, DC 20001
202-336-7016 Fax: 202-408-8072
www.ccsso.org
ISBN: 1-933757-08-6
Copyright © 2007 by the Council of Chief State School Officers. All rights reserved.
Quality Assurance Practices associated with Producing Cohort Graduation Rates
Strategies for Increasing Reliability and Validity of School Accountability Data and Decisions Workgroup
Robin Taylor (DE)-Chair Wes Bruce (IN)
Steve Hebbler (MS) Keith Kameoka (HI)
Pat McCabe (CA) Matthew Pakos (MA) Thomas Spencer (LA) Rachelle Tome (ME) Cathy Wagner (MN)
John Weiss (PA) John Wickizer (KY)
Comments and Suggestions: Deborah Newby, CCSSO; Nancy Smith, Data Quality Campaign; ASR SCASS members and guests
Table of Contents
Executive Summary .........................................................................................................................1
Section 1: The graduation rate issue for states.................................................................................2
1.1: Current graduation rate calculations .............................................................................4
1.2: Why a cohort method?..................................................................................................8
Section 2: What is needed to produce cohort rates? ......................................................................12
2.1: Data collection ............................................................................................................12
2.2: Report production .......................................................................................................19
Section 3: Quality assurance practices in producing cohort graduation rates ...............................23
3.1: Applying QADM to graduation rates .........................................................................25
3.2: Capacity stages ...........................................................................................................28
3.3: Actions prior to public release ....................................................................................31
3.3.1: Internal auditing...........................................................................................31
3.3.2: Data collector verifications..........................................................................32
3.3.3: Rate assignments..........................................................................................32
3.3.4: Trend analysis ..............................................................................................33
3.3.5: Screening Report Templates........................................................................34
3.4: Actions after public release.........................................................................................34
3.4.1: After action reviews.....................................................................................34
3.4.2: Risk assessments..........................................................................................34
3.4.3: External audits .............................................................................................35
3.4.4: On-site monitoring.......................................................................................35
3.4.5: Technical manuals .......................................................................................36
3.4.6: Data collector trainings................................................................................36
Section 4: Conclusion ....................................................................................................................38
References......................................................................................................................................40
Appendix A: AYP graduation rate designs across ASR member states........................................43
Appendix B: Graduation rate design comparisons ........................................................................49
Appendix C: Selected auditing practices of selected states ...........................................................52
Appendix D: Mississippi Department of Education Report ..........................................................53
Appendix E: Graduation rate proxies: Utah and Minnesota..........................................................71
Appendix F: External auditing: Utah.............................................................................................77
List of Tables
Table 1: Proxy limitations summary................................................................................................7
Table 2: Data elements and conditional codes for production.......................................................15
Table 3: Error control issues (school-level)...................................................................................30
List of Figures
Figure 1: Graduation process controls ...........................................................................................17
Figure 2: Indiana Department of Education graduation rates........................................................21
Figure 3: Indiana Department of Education graduation business rules .........................................21
Figure 4: Delaware’s graduation rate diagnostic matrix for SY 2005-06......................................26
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 1
Executive Summary
The Council of Chief State School Officer’s State Collaborative on Accountability
Systems and Reporting (ASR SCASS) has spent two years examining issues of data validity in
state accountability systems. This paper is the second in a series exploring methods of
implementing quality assurance practices in these systems, this one focusing on the calculation
of cohort graduation rates.
As of 2007, states are using a combination of many different methods to determine
graduation rates from local high schools, but most are preparing to move to a cohort calculation,
one that tracks students from entrance to exiting high school. Sensitive to the data requirements
and potential data pitfalls of calculating a cohort graduation rate, Quality Assurance Practices
Associated with Producing Cohort Graduation Rates explores how graduation rates are
calculated in many states, the need for calculation of a cohort graduation rate, and details several
quality assurance practices states can use to produce valid and reliable cohort graduation rates.
Focused on linking the data quality issues to accountability results from a practitioner’s
viewpoint, the paper lays out several quality assurance steps states could implement at various
points. First, states can utilize the expanded Quality Assurance Data Matrix (QADM), developed
for Validity Threats: Detection and Control Practices for State and Local Education Officials, as
a self-evaluation and tool for mitigating error. Other steps to implement both during the
calculation process and after reporting include internal auditing, data collector verifications, rate
assignments, trend analysis, risk assessments, external audits, on-site monitoring, and data
collector trainings. Examples from states are provided to demonstrate the production sequence
and potential reporting formats. Finally, states are reminded to constantly evaluate the efficacy
of their quality assurance practices to ensure they are producing the intended goals.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 2
Section 1
The Graduation Rate Issue for States
The 2001 reauthorization of the Elementary and Secondary Education Act of 1965
(ESEA), known as the No Child Left Behind Act of 2001 (NCLB), heightened data demands of
every state. The law required every state to report disaggregated student assessment results at the
school and district level as well as each high school and district’s graduation rate.
Given the status of state student data systems in 2001, few states or districts were
prepared to provide anything but a proxy rate for their students, estimates of the number of
Graduation rate, as defined in NCLB, 200.19 Other Academic Indicators
(a) Each state must use the following other academic indicators to determine AYP:
(1) High Schools.
(i) The graduation rate for public high schools, which means–
(A) The percentage of students, measure from beginning high school, who graduate
from high school with a regular diploma (not including an alternative degree that is not fully
aligned with the State’s academic standards, such as a certificate or a GED) in the standard
number of years; or
(B) Another definition, developed by the State and approved by the Secretary in the
State plan, that more accurately measures the rate of students who graduate from high school
with a regular diploma as defined above
(ii) In defining the graduation rate, the State must avoid counting a dropout as a
transfer.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 3
students graduating from high school compared to the number beginning high school four years
earlier. There was little capacity to track individual students through their high school careers to
determine when, and if, they graduated, producing a cohort graduation rate.
Since 2001 several national organizations as well as the U. S. Department of Education
(ED) have developed alternative rates to the various state rates used for calculating AYP, such as
the Averaged Freshman Graduation Rate (AFGR), the Exclusion-Adjusted Cohort Graduation
Indicator (EACGI), the NGA graduation rate, and others. As the nation’s education reform focus
shifted to include high school success, a common, accurate calculation of student success in high
school (as defined by graduation in four years) was desired, both to measure the efficacy of high
schools as well as to help target resources to schools in need.
As of 2007, states are using a combination of many different methods to determine
graduation rates from local high schools, but most are preparing to move to a cohort calculation.
Sensitive to the data requirements and potential data pitfalls of calculating a cohort graduation
rate, one that follows students across their high school careers, the Council of Chief State School
Officer’s Accountability System and Reporting State Collaborative (ASR SCASS) began
examining the issues involved with producing valid and reliable graduation rate results in 2006.
The resulting document explores how graduation rates are calculated in many states, the need for
calculation of a cohort graduation rate, and focuses on several quality assurance practices states
can use to produce valid and reliable cohort graduation rates.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 4
1.1. Current graduation rate calculations
One of the most basic questions in evaluating the effectiveness of high schools is whether
students who enter high school as freshman progress through the curriculum and graduate
approximately four years later with a high school diploma. Two fundamental approaches can be
used to answer the question:
1. Use cross-sectional data and estimate, or
2. Establish longitudinal data system and produce differing rates.
Until recently, most states did not have the necessary student-level information system to
accurately monitor student progression across multiple years. State officials used proxy methods
to estimate graduation, completion, and dropout rates based on the most current data available.
Many of these estimates were fraught with data errors produced by user inputs, system default,
and limited quality assurance practices.
It is largely due to technology limitations that many states rely on proxy indicators to
determine graduation rate. (See Appendix A, Table A1 for a description of graduation rate
designs for selected states.) As described by the NISS (2004) study, tracking the progress of
students across time requires, at minimum, a student-level information management system.
These systems must be sophisticated enough to track student movement individually and in
cohorts throughout the state during the current school year and across multiple years.
A student cohort, by definition, is a group of individuals who start at a given point in time
and are then examined again at future intervals. The typical progression is from one grade-level
to the next, thus producing “natural” intervals to measure changes in cohort characteristics.
Students are assigned to progressively higher grades according to those criteria found within
state policy (e.g., high stakes testing) and local pupil progression plans. Students are expected to
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 5
attain certain educational benchmarks for progression into higher grades. For many students,
these criteria are not met during their K-12 education resulting in students being retained in one
or more grade levels, thus exiting their original cohort.
States and districts have made good faith efforts to accurately report graduation data to
the general public. Yet graduation proxy estimates requires several assumptions that become null
and void for particular groups of students. First, students typically earn enough credits to be
reclassified into the next higher grade on an annual basis. Some students will remain at a
particular grade until they have accumulated enough credits to progress. Other students will earn
credits at faster rates, thus completing high school in less than the standard four years.
A second assumption is that students attend high school at a school offering the complete
set of courses or grade ranges. Many districts are reorganizing the high school experience into
several different schools (e.g., 9th grade or career academies). This phenomenon complicates
how one school can be held solely accountable (for NCLB purposes) for the student graduating
on-time.
A third assumption that students’ status is known and accurately reported by local
officials. Students may
(a) transfer from one school and enter another without notifying school officials,
(b) reach an legal age to drop out of the system,
(c) move to another state or country,
(d) be placed in a juvenile center, incarcerated, or become wards of the state, or
(e) exit the system only to reenroll at a latter date.
These conditions along with numerous others require detailed reporting by local officials
and quality assurance procedures.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 6
However, proxy indicators have numerous design limitations when used to address on-
time graduation rates. In some cases, proxies can estimate graduation rates, but rarely at the
school-level. The Mississippi Cohort method, detailed in Appendix D, avoids numerous
limitations by aggregating student-level data to higher units of analysis. This procedure also
produces other rate indicators, such as four and five year completers, along with on-time
graduates. The need for multiple years of data remains an implementation barrier for newly
established (e.g., Oklahoma’s WAVE) or emerging (e.g., Nebraska’s NSSRS) student
information system. These systems will require several years of implementation data before they
will be able to produce graduation rates based on a cohort approach.
Most states have used cross-sectional data to estimate on-time graduation rates. These
cohort-type “proxies” are currently being used to meet graduation rate reporting requirements
prescribed by federal regulations (i.e. 34 CFR Part 200). Unlike Louisiana, Tennessee,
Delaware, Texas and a few others, many states are in their first decade of operating a student
information system able to: (a) assign unique student identifiers, (b) track student movement
over time by using data mining tools (typically within a data warehouse infrastructure), (c)
implement a comprehensive series of exit codes, (d) screen and audit data elements, and (e) build
LEA capacity to ensure data integrity. Without these system components, state-level reporting is
based on aggregate data submitted by LEAs. Dropout estimates were readily used as proxies for
graduation rates (for example, see Appendix D, Utah’s dropout definitions); however, these data
contained numerous anomalies that produced, at best, graduation estimates. Further, few states
apply a cohort method when calculating graduation rates for AYP purposes.
A critical review of these graduation proxies demonstrates clearly their inability to meet
the rigorous design necessary to produce school-level, cohort graduation rates. At best, the
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 7
proxies allow for estimates at the district or state level but even at these units of analysis, critical
production business rules are not defined, reducing the reliability and validity of these
calculations over time. Table 1 summarizes the limitations of various proxy methods.
Table 1. Proxy Limitations Summary
Limitation* AFGR MN HSGR SCPI GGI MSC
Ignores dropouts Yes
No
No
Yes
No
Includes retained 9th-graders Yes
Yes
No
Yes
No
Includes late and early graduates Yes
No
Yes
Yes
No
Includes alternative completers No
Yes
No
No
No
Ignores out migration Yes
Yes
Yes
Yes
No
Ignores in migration Yes
Yes
Yes
Yes
No
Ignores students still enrolled in school No
Yes
No
No
No
Does not represent a 4-year time span No
No
Yes
No
No
* Adapted from Seastrom et al. (2006) ** Average Freshman Graduation Rate [AFGR]; Minnesota High School Graduation Rate [MN-HSGR]; Swanson Cumulative Promotion Indicator [SCPI]; Greene Graduation Indicator [GGI]; and Mississippi Cohort [MSC]
Perhaps the most significant business rule missing in proxy methods for determining
graduation rates is associated with student mobility. Typically described as “migration,”
students moving into or out of the cohort must be addressed. Simply excluding these data will
result in an ever growing number of students, whose graduation status is never accounted for in
public reporting or AYP determinations. This phenomenon has some similarities with the full
academic year (FAY) provision that allows student performances at the school-level to be
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 8
removed if they have not attended the entire school year. Yet, these business rule limitations and
differing report methods do not equate to students not being accounted for within the education
system.
1.2. Why a Cohort Method?
A cohort approach to calculating graduation rates can provide clear and concise
information on how many high school freshmen complete their educational experience and
graduate within a specific period of time. Unlike other approaches that estimate the number of
students who complete their high school experience, the cohort approach makes these
determinations for each student. This is accomplished by examining multiple years of data and
tracking the student’s enrollment and exit information.
Since data is tracked at the individual student level, the cohort method allows for both
data aggregation, analysis at the district, state, or national level, and data disaggregation, data on
the graduation rate of student subpopulations at the school level. Detailed characteristics, such as
student mobility, retention, drop-out, and other factors can be examined at each level. Contrast
this to proxy methods, which generally produce data at the district or state level and cannot move
to lower units of analysis. This limitation was obvious when federal requirements outlined in
NCLB required using graduation rates for AYP determinations.
In 2005 the National Center for Education Statistics (NCES) at the United States
Department of Education announced a new calculation of graduation rates for each state. Called
the Averaged Freshman Graduation Rate (AFGR), the calculation uses existing data submitted
by states to NCES through the Common Core of Data. Essentially, the percentage of graduates is
figured as follows:
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 9
The calculation also allows for ungraded students to be allocated across grades. The
authors point out that though this rate is “not as accurate as an on-time graduation rate computed
from a cohort of students using student record data, this estimate of an on-time graduation rate
can be computed with currently available data” (Seastrom, M., et al., 2007). However, based on
the calculations available for cross-state graduation rate comparisons, this was considered the
best alternative “based on a technical review and analysis of [other possible] estimates”
(Seastrom, 2007). The AFGR for all states for 2002-03 and 2003-04 was published in 2007,
though it is possible to figure the rate for several previous years based on the NCES published
formula.
The National Institutes of Statistical Sciences/Education Statistics Services Institute’s
Task Force on Graduation, Completion, and Dropout Indicators published its final report
(NISS/ESSI, 2004) on technical issues associated with calculating graduation (also completion
and dropout) indicators. A task force comprised of researchers, state and federal officials, and
others produced a series of recommendations to accurate calculate an on-time graduation rate.
The group reported the following:
1. No single indicator of graduation, completion, or dropouts can serve all purpose;
2. No indicator allowing for “exclusions” can be free of perverse incentives;
3. Uniformity across states in reporting (along with methods for calculating
indicators) graduation, completion, and dropout indicators;
4. Data quality and availability; and,
5. Verification of interstate transfers.
[(students in 8th grade in year y-1) + (9th grade in year y) + (10th grade in year y+1)]/3 graduates with regular diplomas in year y+4
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 10
These findings were reported in conjunction with a recommendation that the graduation
indicator must have three key properties: (a) data are derived from student-level collections and
are functions of entry and year of graduation, (b) entry and graduation year is in cumulative
form, and (c) exclusions are defined by a narrow set of business rules. These principles
supported the creation of the Exclusion-Adjusted Cohort Graduation Indicator (EACGI), which
has now been adopted by the NGA. This cohort approach uses student-level data and a series of
data elements and conditions that records student status across multiple years.
In 2005, the nation’s governors, through the National Governor’s Association, adopted
the EACGI rate, represented as follows:
Governors from all 50 states agreed to work toward calculating the rate, adjusting data
systems to collect the necessary indicators if necessary. This calculation is referred to as the
“NGA Graduation Rate.” This formula applies only to student who receive standard diplomas.
In 2006, NGA and CCSSO convened a technical panel to create technical guidelines to
allow this calculation to be utilized in all states, realizing the governors’ aim of a comparable
graduation rate utilizing high-quality, student-level longitudinal data. A selection of the panel’s
recommendations follow:
1. Calculate the 4-year rate, as specified in the NGA rate formula, and also a 5-year
graduation rate.
2. Define, document, and implement a detailed student-level exit data collection process
to account for students who leave the public school during or between school years.
On-time graduates by year X [(first time 9th graders in year X-4) + (transfers in) – (transfers out)]
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 11
3. Establish a process by which the state educational agency (SEA) reviews statistical
trends of exit data within and across school years to identify potentially erroneous
data.
4. A student whose IEP allows an extra year to graduation, or who is receiving services
for limited English proficiency who is allowed extra time to graduate, should be
placed in the cohort with which that student is expected to graduate.
5. States should clearly describe by component how they are calculating the rate.
(Smith, 2006).
Many states redoubled their efforts to develop systems to allow them to figure the cohort
graduation rate following the NGA agreement. At the same time, many continued to find
themselves reeling form the sheer quantity of data generated and required by NCLB assessment
and data calculations. A breakdown in the reliability of the data at any point from input at the
school level to data transfer at the district or state level, to calculations, could produce spurious,
or at least unreliable, results for any of these cohort calculations.
Though each state education agency is working to follow each individual student
throughout his or her high school career, in order to report when and if successful high school
completion takes place reporting cohort graduation rate calculations, agency staff need to ensure
1. the proper tools are in place to collect information necessary to complete the
calculation, and
2. make sure the quality assurance practices are in place so that the reported figures are
valid and reliable.
The second part of this paper focuses on these two topics.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 12
Section 2
What is needed to Produce Cohort Rates?
There are three fundamental activities necessary to produce cohort graduation rates in a
manner consistent with the NISS or NGA recommendations. These activities are to: (a) collect
data across multiple years, (b) create business rules and design logic for production, and (c)
implement quality assurance practices. Within each of these macro-level activities are numerous
tasks requirements for both local and state-level officials.
Data inputs describing student characteristics, enrollment and exit status, and other
information must be acquired from the data collectors. Local and state information managers
must implement data integrity and verification processes, while policy makers must establish the
overarching context, business rules, and design logic used to report targeted results. The
production process requires migration of data into the programmer’s code, execution of the code,
and validation checks before finalization. Once finalized, the data are placed in the report
production cycle and disseminated to data collectors as required by state and federal policies and
procedures.
During the entire work cycle, quality assurance practices used to verify and validate the
data are continuously being implemented. These data are used to correct errors, reduce the
probability of their occurrences, and verify alignment between the underlying data and that
reported. These numerous activities must adhere to a rigid, standardized set of operating
procedures or run the risk of producing inaccurate information.
2.1. Data collection
Perhaps the most critical infrastructure necessary to produce cohort graduation rate is a
centralized student information system (SIS) that uses a unique identification system to identify
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 13
students. Without a data system that can accurately track a student’s entry, progression, and exit,
no calculation of cohort rates will be accurate.
Control efforts within the SIS must have a comprehensive set of internal verification
processes to detect and correct irregularities such as duplicate records, missing data fields,
illogical data, and invalid enrollment and/or exit codes. These error mitigation efforts are
typically done prior to score production because they are used to establish the master file set.
Validation efforts in some states, such as Nebraska, Delaware, and Louisiana, use quantitative
approaches to examine the reasonableness of the data. Data cells identified as beyond
established thresholds are “flagged” for qualitative review. These validation activities in other
state settings can range from informal reviews done by internal staff to comprehensive auditing
by third-parties.
Business rules: Front loading investments in human capital
Verification and validation actions must be carefully planed to ensure limited fiscal
and human resources are effectively allocated while adhering to established production and
reporting schedules. State and local officials must not only develop and maintain the
necessary hardware and software infrastructure, but also must direct investments toward
human capital development. “Front loading” their investments in the personnel (data
collectors) who will enter, review, and validate data indicates a proactive strategy to improve
data quality. In other words, improvement efforts should seek to maximize resource use by
focusing on high risk data elements and production processes requiring human interactions.
For better or worse, data entry and upkeep is ultimately up to individuals, data collectors at
the school or district level.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 14
The “human factor” (HF) can contribute to error production and its removal during each
sequence of the production phase. The first HF must provide information detailing student
characteristics such as the school they are attending, their assigned grade, and the types of
programs they participate in. The HF is complicated when the data collectors use complex
business rules and IT taxonomies (NFES, 2006) in which they may have never received training
or inadequate technical support. Data are provided by data collectors who must assume
responsibility for their accuracy. This responsibility may be shared across several
program/school staff (e.g., attendance officer, counselor, front office secretary, local IT manager)
and readily results in data being duplicated or modified without knowledge of all responsible
parties. Many of these data remain static over time or change in highly predictable patterns. For
example, gender and date of birth are clearly fixed characteristics that do not change over time,
yet school assignment can readily change within any given school year.
Students contribute to the HF by changing schools, attending unique programs, becoming
incarcerated or progressing from one grade to the next in atypical ways. Data collectors update
the records of these students using the available codification within the information system. This
information can require updates within a given school year or across years, especially with
highly mobile students.
Data input errors entered into the early stages of graduation rate score production will
obviously produce spurious results. Data verification and automated edit checks of data
elements like those provided within Table 2 will correct much of the faulty information before it
enters into the production phase.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 15
Table 2: Data Elements and Conditional Codes for Production
Data Element Description Needed for Cohort
District ID Name and NCES code Yes School ID Name and NCES code Yes School Year Year associated with data Yes Student Name First, middle, and last name No Student Age Chronological age No Student ID Unique identifier Yes Grade Assigned grade including “ungraded” programs members Yes Gender Student’s sex No-but Ethnicity Student race No-but ED Economically disadvantaged flag No-but SWD Students with disabilities flag No-but LEP English language learner flag No-but Migrant Migrant flag No Title I Title I flag No Resident School School assigned for accountability No Resident District District assigned for accountability No Enrollment Date Entered school Yes Enrollment Type Public, Non-public, Private, Attending in another state, Resident
attending non-public school Yes
Exit Date Left school Yes Exit Reason Reason codes clusters (NFES, 2006)
1. Still enrolled 2. Transferred 3. Dropped out 4. Completed 5. Not Enrolled, eligible to return 6. Exited-neither completed nor dropped out
Yes
Completion Type Types of diplomas Graduate with diploma, Industry based certificate, GED only, Certificate of Achievement
Yes
Cohort Assigned Cohort identifier Yes
The second HF are the activities associated with the constructing of student cohorts.
Student cohorts are developed through a series of data manipulations and linking across differing
years. Although this is accomplished through computer software programs, it requires the
specialized skills of humans to write, manipulate, and execute the code. Validation checks used
to produce error reports and/or to reject data outside of prescribed tolerances must also be
conducted by staff members. Data quality checks by information managers may result in
submitted data being changed in the master files. For example, a student may have an exit date
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 16
Once created, the programming code is manipulated to reflect changes in policy,
easily detailed for clarity in flowcharts.
and condition showing him/her as a dropout; however, a valid enrollment date is identified
keeping the student within the cohort used for the district but not at the school-level. These types
of record corrections are also needed in assigning and reassigning students to particular cohort
groups. Enrollment and assigned grades are needed to determine if the student is repeating the
grade, although these data may be collected using a customized data element. For example,
students who graduate prior to the standard year must be maintained in their original cohort, so
students who do not progress to higher grades can be identified.
The third HF occurs within the report production cycle. Report production processes
require detailed business rules articulating how data are to be manipulated, errors resolved,
variables created, and other operational tasks executed. Flowcharting the decision logic prior to
software programming assists in graphically organizing the decision being executed within
underlying software code. This simple technique is a helpful tool in linking the programming
and policy logic as the lexicon for each system is very different. Modularizing the scoring code
can quicken internal auditing and other validating activities, while reducing the time necessary to
change and test additional coding. The results must undergo a series of quality assurance checks
to verify the results meet the specifications outlined in the business rules and design logic. These
validation checks can range from simply examining the frequency distributions across differing
units of analysis to conducting score replication by external entities.
In Louisiana, and other states, preliminary graduation (and dropout) data are provided to
LEAs prior to report production. This process allows local officials to evaluate outcomes from
their score production processes and correct data errors. Once reviewed, the final data is
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 17
migrated into the reporting production sequences used to manufacture information for the
selected audiences. Customized reports may be required for improvement efforts or to conduct
auditing functions. Overall, the HF exists throughout the graduation process reflected in Figure
1. Efforts to ensure reported data are accurate and credible requires addressing the HF’s
influence within the system.
Figure 1. Graduation Process Controls
Graduation Process Controls
Input Cluster A
Student Characteristics[Demographics &
Program Membership ]
Input Cluster B
Student Status[Enrollment &
Completer]
Input Cluster C
Student Cohort[Assignment &Reassignment]
Public Reports
AYP
StateAccountability
NGA
Strategic Plan
End
Use
r
IT
After PublicRelease
On-SiteMonitorings
End-User Trainings
External Audits
Technical Manuals
Risk Assessments
After ActionReviews
IT Validation
*Field Specs*Error Report
*ID Assignment
Screening
* Duplicates*Enrollment Status*Completer Type
*Transfers*Missing
*Cohort Year
Post Production QA
Rate Assignments
Trend Analysis
Screening ReportTemplates
End-User (LEA)Verifications
Internal Audits
Rate Production
Business RuleApplication
Policy Alignment
Decision andOperational Logic
End
-Use
r
The production of graduation rates requires a series of complex data manipulations and
computations. To illustrate this point, the procedures used by the Mississippi Department of
Education to calculate cohort graduation, completer, and dropout rates (see Appendix D for the
complete February 12, 2007 report) are presented in the excerpt below.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 18
Background This information provides details on the procedure developed for tracking a full cohort of students in the
Mississippi Student Information System (MSIS) and calculating estimated dropout, completion, and graduation rates for the full cohort. The objectives were to ensure that all students would be included in an appropriate cohort (i.e., no students would be systematically left out) and to use data available in MSIS to calculate accurate counts and rates for dropouts, completers, and graduates.
Since MSIS was implemented statewide at the beginning of the 2001/2002 school year, the first four-year full cohort could be tracked using data from the end of school year 2004/2005 updated with the results of the 2005 summer activity procedure. All the required data were available in April 2006. Using the most appropriate analysis techniques, there are still certain errors that will be present in the calculated counts and rates. Those include coding errors in the data transmitted to MSIS from the district student administrative packages (SAPs), a small amount of data inconsistency (generally in the first school year) due to edit checks that were implemented later, and students whose final disposition is unknown because they left the system during summer 2002 or summer 2003 (before the summer activity procedures were implemented).
*Technical Note: The cohort includes retained students. The Mississippi Department of Education is
currently working on the next cohort (beginning with 9th graders in 2002/2003) and will be able to include in the full cohort only the first-time 9th graders. Rates reported in this initial work will differ from future values excluding repeaters.
Decision Logic The decision and programming logic for implementing a full cohort tracking system was approved and
data files were built using the data in MSIS. The steps used for tracking a cohort and analyzing the data are as follows:
1. All students who entered MSIS as ninth graders during school year 2001/2002 were identified and their data written to a data file. For each student, a variety of data variables were extracted from MSIS, including the last know disposition for students who were not still enrolled somewhere in Mississippi at the end of 2004/2005. [N=41,160]
2. All students who entered MSIS as tenth graders during school year 2002/2003, and were not already in the data file, were identified and added to the file. [N=4,384]
3. All students who entered MSIS as eleventh graders during school year 2003/2004, and were not already in the data file, were identified and added to the file. [N=2,344]
4. All students who entered MSIS as twelfth graders during school year 2004/2005, and were not already in the data file, were identified and added to the file. [N=1,304]
5. All students coded grade 56 or 58 (self-contained special education) who were the age of typical ninth graders during school year 2001/2002 were identified and their data written to a separate data file. [N=1,310]
6. All students coded grade 56 or 58 (self-contained special education) who were the age of typical tenth graders during school year 2002/2003, and were not already in the data file, were added to the file. [N=452]
7. All students coded grade 56 or 58 (self-contained special education) who were the age of typical eleventh graders during school year 2003/2004, and were not already in the data file, were added to the file. [N=331]
8. All students coded grade 56 or 58 (self-contained special education) who were the age of typical twelfth graders during school year 2004/2005, and were not already in the data file, were added to the file. [N=164]
9. Student records in the regular grade file and the grade 56/58 file were matched by MSIS ID to identify any duplicates. Of the 2,257 grade 56/58 students, 58 students had records in both files. An examination of the duplicate data records revealed that the only difference was in the cohort flag (year of entry into the cohort). Therefore, a procedure was run to eliminate one of the duplicate records and merge the regular grade and grade 56/58 data files. Unduplicated records comprised the full cohort. [N=51,391]
10. A tentative solution was used for student data records that contained both a completion code and a transfer code. This problem reflected data from the earliest year, prior to the addition of certain edit checks in MSIS. The solution was to blank out the transfer code for any student with a completion code (T, O, G, OD).
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 19
11. The cohort flag, transfer codes, and completion codes in the data file were used to set unique binary (0,1) variables that could be used for aggregating the student data at the school, district, and state levels.
12. The full cohort data file was matched by MSIS ID to the Month 1 2001/2002 enrollment file to determine which students coded as ninth graders in 2001/2002 were "true beginning" cohort students rather than students that had entered later in the school year. Of the 41,160 students, 38,833 were true cohort students and 2,327 had been added.
13. The full cohort data file was matched to the Month 9 2004/2005 enrollment file to determine the status for students who had no transfer code and no completion code. There were 145 "lost" T1 and T2 transfer students and 2,961 students enrolled in some grade.
14. For students in the full cohort data file whose disposition was still unknown after the above steps, summer activity codes were applied. Of the 2,741 students with summer activity codes from 2004 or 2005, 6.6% had completed all requirements except for a passing score on one or more tests needed for graduation, 58.5% had been coded as dropouts, and 35.0% had been coded as transfers or deaths.
15. Ultimately, there were 4,648 students in the full cohort data file whose final disposition was unknown. Since they were probably students who were lost during summer 2002 and summer 2003 (prior to the implementation of the summer activity process), it was decided to apply apportioning constants to the aggregate counts based on the actual percentage of students coded as summer dropouts and transfers during 2004 and 2005.
16. All the full cohort data records were summarized (aggregated) at the state level based on tentative logic outlined in the steps that follow.
17. Binary values (0 or 1) were accumulated across all student records for each of the following variables. Total Full Cohort N-Count, Cohort 0 (true cohort), Cohort 1, Cohort 2, Cohort 3, Cohort 4, T1, T2, T3, T4, T5, T7, T8, Z1, IS1, IS2, IS3, ST4, ST5, ST7, ST8, SZ1, Dropout, Dropsumm, Diploma, Trad, Occu, GED, Cert, ABT, Stillenrl, and Unknown.
18. The unknown student aggregate count was multiplied by 0.585 (58.5%) to yield the statewide number estimated to have been summer dropouts (SD1 through SD21).
2.2. Report Production
Reporting graduation rates has continued to be a challenge for national, state, and local
agencies. One complexity is in defining the audience receiving the information, then
customizing the reported data in a manner that meets the data collectors’ needs. The general
public typically receives information about graduation and dropout rates from local media
outlets. These data may or may not accurately reflect the official information provided by the
state agency. Explaining the plethora of business rules, differing methodologies, and reporting
formats often results in misinformation being disseminated. In worst case scenarios, politicians
and other policy-makers act on spurious information and make inappropriate causal links
between policy goals and reported outputs.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 20
The passage of NCLB and its mandated reporting requirements has increased the
communication burden of state officials. Multiple graduation definitions and disaggregating
graduation data of subpopulations has resulted in many states using web-based decision-support
systems (DSS). Decision-support systems typically afford data collectors the ability to develop
customized queries for comparative purposes. In their simplest forms, the public can download
static data tables in a format that can be manipulated using commercial software packages (e.g,
Excel, SPSS, SAS). More sophisticated DSS allow the data collectors to define data elements
and extract files from the data warehouse to produce longitudinal data sets. Query-building tools
allow the user to link student data across multiple years without providing personally identifiable
records. In Indiana, the Indiana Department of Education has created a series of graduation and
associated data figures and charts for each of its schools and districts. These displays (see Figure
4) provide data collectors with easily understandable statistics on graduation rates based on
cohort calculations. The data for other types of students such as (a) dropouts, (b) GED
recipients, and (c) students still enrolled are compared to the percent of on-time graduates. The
state further disaggregates the graduation data by those reporting categories outlined by NCLB.
Additional information is provided to the data collectors by moving their cursers over the
selected data element, while hyperlinks connect data users with the applicable business rules (see
Figure 3).
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 21
Figure 2. Indiana Department of Education Graduation Rates
Figure 3. Indiana Department of Education Graduation Business Rules
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 22
It is clear that some states have the robust data systems necessary to produce cohort
graduation rates, and more. The tools described above allow for in-depth analysis not
possible with proxy graduation calculations. However, especially with the vast quantity of
publicly-reported data, as in the Indiana example, it is necessary to institute layers of quality
assurance practices to ensure the data generated is valid and reliable.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 23
Section 3
Quality Assurance Practices in Producing Cohort Graduation Rates
Quality assurances practices are critical in producing valid and credible graduation rates.
Whether the agency is reporting the NGA, cohort, status, and/or completion rates, the processes
used to examine data inputs prior to score and report production is essential.
Quality assurance practices in most states are a combination of externally reported data,
internal review procedures (field specifications and error reports), and data verification
techniques. These processes allow agency officials to assert their graduation and dropout
determinations are valid representations of events in local schools and districts.
The judicious application of control measures (Wheeler & Lyday, 1989) is one approach
within a quality assurance design. Control procedures are used to evaluate targeted data
elements during the production cycle and mitigate unwanted error. Without such control
procedures, valid inferences about performance cannot be made for the given year. The controls
must be sensitive to detect slight changes in graduation (including non-graduate) indicators,
while discerning actual change from natural variability and non-systematic error.
Progress in creating clear and concise operating rules has been made at the national and
state-level. Graduation, completer, and dropout data being collected and reported from student
information systems has significantly improved the quality of the results. States are developing
and implementing quality assurance practices at critical points in the score development process.
Some states, such as Indiana, Utah, and Delaware, are continuously examining their internal
practices and capacity used to produce graduation rates. Other states, such as Mississippi and
Louisiana, conduct extensive quantitative design testing prior to policy adoption.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 24
All state data contain data anomalies; some are valid while others do not represent actual
facts. Several critical areas, such as primary keys used to link multiple years of data for a
student, require differing levels of error detection and controls. State and local officials have
limited time and human resources to investigate each and every data point in their information
systems. However, some data elements require more effort than others because of their overall
influence on the final score. A better understanding those critical data associated with applying a
cohort approach can assist state and local leaders directing quality assurance efforts.
A state applies quality assurance practices to document calculations for external review
Delaware’s schools and districts operate within the public sector; thus their activities
must reflect both transparency and stakeholder input. Accountability activities, part of the
greater education system, must ensure that scarce resources are being used in the best interest
of the public. Continuous improvement practices at both the state and district-level strive to
maintain high quality through transparent practices and credible decision-making to promote
consumer satisfaction. One approach used by the Delaware Department of Education (DDOE)
is to document the entire accountability subsystem (DDOE, 2006), including the production of
graduation rates, in sufficient detail so as to facilitate external reviews. Recognizing how the
accountability system is dependent upon information management and assessment subsystems;
the agency must annually examine its current practices. Anticipating having to calculate the
National Governors Association’s (NGA) graduation rate and other completer information,
state officials evaluated areas needing additional capacity development. Because of the
interdependency among subsystems within the agency, specifically information management
and accountability programs, special consideration was been given to understanding how
improvement efforts will create spillover effects (i.e., additional benefits at no additional cost).
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 25
3.1. Applying the QADM to Graduation Rates
The Quality Assurance Diagnostic Matrix (QADM) was designed through research for a
previous paper on data validity issues for CSSSO’s ASR SCASS, and presented as a tool for
SEAs to examine the quality assurance capabilities in producing cohort graduation rates. The
QADM was derived from the conceptual work found within the Systems Security Engineering
Capability Maturity Model® (Carnegie-Mellon, 2003) and Six Sigma’s DMAIC® process
(Pyzdek, 2003). The QADM for calculating graduation and dropout rates comprise a set of
quality assurance indicators for two critical educational subsystems: (a) information
management, and (b) accountability [including other public reporting]. These performance
indicators are aligned within each of the targeted subsystems and evaluated across a capacity
continuum. The continuum is organized into four stages that reflect increased organizational
capacity and quality assurance sophistication.
The performance indicators within each subsystem represent a finite set within the
universe of potential indicators. For use in graduation and dropout rate production, several
indicators were modified from the original design (CCSSO, 2006). The graduation rate
indicators focused on validating data, training data collectors, improving quality practices, and
evaluating system controls. This set would most likely be refined by state and local officials to
match the operational activities unique to each subsystem.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 26
Figure 4. Delaware’s Graduation Rate Diagnostic Matrix for SY 2005-06
Quality Assurance Diagnostic Matrix (QADM)-2006 Stage 4 4. Leading
3b. Streamlining Stage 3 3a. Improving Pr
oact
ive
2b. Documenting Stage 2 2a. Standardizing
1b. Reviewing Stage 1
1a. Planning
0b. Completing
Rea
ctiv
e
Stage 0 0a. Omitting
E
duca
tiona
l Sy
stem
s •
IT (1
.1 th
ru 1
.5)
• A
ccou
ntab
ility
(3.2
thru
3.
5)
1.1-
Inte
grat
es p
olic
y ch
ange
s
1.2-
Val
idat
es p
rogr
am
mem
bers
hip
1.3-
Trai
ns d
ata
prov
ider
s
1.4-
Exam
ines
fiel
d sp
ecs a
nd
asso
ciat
ed b
usin
ess r
ules
1.5-
Det
erm
ines
dat
a qu
ality
ne
eds
3.2-
Ope
ratio
naliz
es p
olic
y
3.3-
Supp
orts
acc
urat
e in
terp
reta
tions
3.4-
Alig
ns sy
stem
co
mpo
nent
s
3.5-
Eval
uate
s TO
A
beha
vior
alch
ange
s
The QADM reflects the capacity stages at a specific moment in time, rather than a
forward looking (goal setting) approach. The vertical axis provides a continuum of quality
process characteristics organized into stages. These stages range from quality practices not being
addressed (Stage 0) to standardized procedures used to maximize efficiencies (Stage 4). The
rating process requires the users to rate each quality indicator found across the X-axis using the
capability scale reflected on the Y-axis. The Y-axis continuum reflects the degree of capacity
necessary for task implementation (see Figure 4).
The first component of the QADM, on the X-axis, requires the agency to select targeted
indicators associated with programmatic aspects of the agency. Within these subsystems, time
and money resources are used to produce targeted outputs. Often several subsystems must work
on individual parts of the targeted output. For example, statewide assessments used student-level
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 27
information to determine critical administrative tasks necessary to produce test scores for each
eligible student. Critical quality indicators are organized within two key subsystems:
Information Management and Accountability. It should be noted that these indicators would
need be converted into operational terms (objectives, tasks, resources, milestones, measures, and
performance standards) prior to implementation (see Maine Department of Education’s Quality
Assurance Plan, 2006).
Information Management
This subsystem comprises the following quality indicators:
• Integrates policy changes into the student information system.
• Validates enrollment and program membership with data collectors and coordinates
information within the agency.
• Trains data providers on error detection using screening and auditing functions.
• Examines field specifications against business rule narratives.
• Determines data quality needs and implements improvement activities.
Accountability
This subsystem comprises the following quality indicators:
• Validates score and report production.
• Operationalizes and standardizes policy.
• Supports accurate interpretations.
• Promotes credible judgment supported by additional evidence.
• Aligns system components.
• Evaluates Theory of Action (TOA) behavioral changes.
3.2. Capacity Stages
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 28
The second component of the QADM is the continuum of quality capacities, represented
on the Y-axis of Figure 4. These capacities are dependent on the time and human resources
available to the entity under review. External demands (e.g., NCLB) create additional resource
demands that otherwise could be directed to improve existing operations. This “Catch-22” has
been a significant challenge to both state and local agencies as they attempt to implement new
programs prior to establishing sufficient capacity for those activities currently operating. In the
QADM, the capabilities are represented as follows:
Stage 0
0a. Omitting - Not addressed by the entity (we don’t do it here)
0b. Completing - Informal and random (someone does some part somewhere)
Stage 1
1a. Planning - Planned but undocumented (someone does it, but it is not in writing)
1b. Reviewing - Monitored but undocumented (someone does it, it [the outcome of the review]
is in a written report but not in the process)
Stage 2
2a. Standardizing - Mixed formality (done, written reports, some documentation and
standardized procedures)
2b. Documenting - Formal yet compartmentalized (done, standardized, documented, subsystem
specific)
Stage 3
3a. Improving - Standardized and validated (done, standardized, improved upon, checked,
subsystem specific)
3b. Streamlining - Systemic and dynamic (done everywhere, continuously improved upon)
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 29
Stage 4
4. Leading - Efficient and transparent (how we do business, industry-leading practices)
The QADM modified for graduation and dropout rates provides a simple graph that
defines the quality assurance needs across the agency. At this point, officials are able to
visualize quality assurance areas needing development and/or augmentation. In Maine, the state
officials at the Department of Education tailored the QADM approach in developing their quality
assurance plan (MDE, 2006). Beyond modifying the indicators, the Maine Department of
Education created performance descriptors reflecting the differing stages of capacity.
Accounting for every student within a state is a complex task made more difficult by
diverse programs, student mobility, changing policies, and political demands. Graduation
information continues to provide an indicator of educational quality and is readily understood by
the general public. The underlying data and the differing methods used to produce graduation
statistics is in of itself complex and diverse.
Three basic graduation rate components are demographic, membership, and
enrollment/exit data. Among the specific data elements within these components, errors can
occur that can have a significant influence on determining the graduation status of the individual
and when aggregating data to higher units of analysis. Errors can be created by either data
collectors, exit code taxonomies (see National Forum on Education Statistics, 2006 publication
for additional information), or a combination of both. Even with the best quality assurance
mechanisms, errors will remain in the master data file used for score production. Other errors
enter into the system because of the complexity of including the information using the existing
business rules. Table 3 outlines some of the technical issues states must address when
calculating cohort and NGA graduation rates. Issues are rated by the potential influence on the
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 30
reported graduation rate. A rating of 1 means data with this type of error produces minor errors
in the overall results. Issues with a rating of 2 contain readily detectable mistakes in the final
scores, while a rating of 3 invalidates the results. For example, Issue #13 would be rated a 2 for
calculating completion rates and a 3 for the NGA graduation rate. The decision logic being that
adding the GED data into the NGA rate is a substantive violation of the business rules. For
completer data, GED data would most likely be a reported category, thus errors would be readily
detected by simple visual inspection of the data.
Table 3: Error Control Issues (School-level)
Potential Impact Issue Completer Rate NGA Rate
1. Changing school grade configurations 2 2 2. Alternate learning centers 1 1 3. Retained students 1 3 4. Summer dropouts 1 1 5. Unknown status 2 2 6. Nongraded students 1 1 7. Juvenile centers 1 1 8. Redistricting 2 2 9. Multiple feeding patterns 1 1 10. “Super seniors” 1 1 11. Transfer to non-public school 2 2 12. Transfer out of state 1 1 13. Transfer to GED program 2 3 14. Transfer to home school 2 2 15. Grade 8 summer dropout 1 1 16. Residence programs 1 1 17. Regional schools 1 1 18. Grade skipping 1 2 19. Enrollment grade is less than prior year 2 2 20. Duplicate records 3 3 21. Completion date precedes transfer data 1 1
1 = Minor errors 2 = Detectable 3 = Invalidates results
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 31
3.3. Actions Prior to Public Release
The quality of any product or service must adhere to minimum quality standards while
seeking ways to improve either quality and/or efficiencies associated with production. In
examining the production processes necessary to report cohort graduation rates, state agencies
must examine those controls being used in other production processes. Controls are those efforts
used to identify and reduce unwanted error. As outlined in Figure 1, quality control can be
organized into three general areas: (a) those associated with verifying score accuracy (post
production), (b) internal reports created within the agency, and (c) reports and activities external
to the agency. Appendix C provides a sample of quality assurance practices for several states.
3.3.1. Internal Auditing (IA)-Internal auditing typically explores its use of screening
standards, production controls, and pre-release quality assurance efforts. The QADM can
function as a tool to accomplish the “where are we now” task. High risk areas and those data
elements within particular schools and school districts that were beyond established tolerances
are flagged for further IA activities. Automated subroutines are written, disseminated, and their
findings are reviewed within the agency. These results may constitute policy changes, additional
control development, or the use of external activities (i.e., on-site monitoring) prior to
accountability score production.
Internal auditing tasks are also focused on the data collection, validation, and production
procedures. These activates are almost always conducted by a third-party not affiliated with the
agency. The findings are typically used to formulate improvement efforts and build staff
capacity. Costs associated with IA activities are moderate to high regardless of whether out-
sourcing approaches are used. Timing is critical for IA activities, especially if it is associated
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 32
with auditing programming codes used to extract, validate, and produce scores. The importance
comes from having IA results available in enough time to implement corrective actions.
The IA tasks use test samples produced from both random and purposeful sampling
techniques. Beyond random selections, a conscious effort is made to include large and small
schools, new schools, and idiosyncratic situations. The typical sample represents approximately
10% of all public schools and districts within the state. When score anomalies are encountered,
additional schools are checked and rechecked until the internal auditor and quality assurance
team member are satisfied that the problem has been corrected.
3.3.2. Data Collector Verifications-Data collectors, those inputting the information and
those persons receiving the final reports are often some of the best resources available to state
officials. Having these individuals support data clean-up and validation activities can
significantly reduce undetected errors. For agencies operating decision support system (secure),
local staff can examine very granular data points at the student-level and make any necessary
corrections. Advanced data validation and score examination tools can be made available to data
collectors attempting parallel data runs.
Preliminary data can serve as a verification benchmark prior to final score acceptance and
public release. Data errors and programming inconsistencies are readily detected by individuals
who have the ability to independently produce their own graduation rates. However, training
costs are high for these activities. Standardized procedures amenable to third-party replication
and robust professional development initiatives are needed for even mature systems. Costs can
be reduced by manipulating existing subsystems and modifying training materials currently in
operation.
3.3.3. Rate Assignments- Examining the graduation, completion, and dropout rates can be
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 33
accomplished using simple comparative checks to validate that the results appear “reasonable.”
This is typically done when prior experiences and knowledge of schools and districts confirm
pre-established expectations. Time and labor costs are relatively low, the technique detects minor
variances within high risk data elements. Newly emerging systems should not rely exclusively on
this approach. Further, state officials would benefit from placing analytical emphasis to each unit
of analysis.
3.3.4. Trend Analysis- A standardized approach used extensively in the manufacturing
and petrochemical markets is the application of statistical process controls (Pyzdek, 2003,
Wheeler, 2003) that monitor variances during production. Delaware (DDOE, 2006) applied this
principle by establishing prevalence thresholds based on several data points. Evaluating the
production results, data elements, and screening results using standardized thresholds allows for
automated quantitative analysis. This approach provides critical information about whether the
underlying data are of sufficient quality to provide credible results.
The trend analysis approach does require multi-wave data, four or more years of data and
the additional time (often six months or more) to finalize the “standard year window.” Thus is
often limited to more “mature” systems. Time and labor costs to establish performance
thresholds, create the necessary programming code, document the procedures, and train staff on
For newly emerging system, another approach is to examine the year to year
matriculation and dropout rates. Over a four-year period the accumulative percentage moving
will always be greater than the amount estimated by a four-year graduation rate. The
difference between the two indicators increases inversely with the proportion of students who
fail to continue to the next grade.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 34
the how to implement methodology are typically high. Modifying procedures used in other
educational subsystems (i.e., assessment) combined with outsourcing are ways to reduce costs.
3.3.5. Screening Report Templates- Screening paper, web-based and oral presentation
materials prior to public release are essential controls for systems of all levels of maturity.
Internal, external or a combination of controllers evaluate the data and narrative language to
ensure it is accurate. Many entities develop simple checklists and spreadsheets that guide the
reviewers’ efforts, while concurrently providing the necessary documentation for future quality
assurance activities.
This approach assumes the underlying data quality and production results are within
acceptable tolerances. Costs in developing standardized procedures and instrumentation is
moderate to low. Implementation costs are moderate and can be reduced by increasing internal
capacity. The resulting efforts reduce the likelihood that senior officials will present information
and supportive background data that differs from the actual production results. Further, it
provides assurances that the publicly disseminated, and data typically reported by the media, are
credible and able to withstand public scrutiny.
3.4. Actions after Public Release
3.4.1. After Action Reviews (Reflections) - This qualitative process is conducted with the
key operational staff, including programmers, policy, public affairs, and accountability
personnel. The basic intent is to highlight the work flow and task accomplishments that worked
as designed and those that failed and required additional time and effort to resolve.
3.4.2. Risk Assessments - Risk assessments are conducted on data elements and
production procedures in which small errors can significantly degrade output quality. These
types of analysis can range from informal meetings within the agency based upon experience to
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 35
formal reports by a risk management entity. All production procedures have some degree of
risk. Risk, the potential for reporting information that does not reflect truth, must be addressed
after each production sequence. Information from these tasks is often used to initiate other
control activities such as on-site monitoring and training. Time and labor costs are typically low
in most situations.
3.4.3. External Audits - Auditing as a control is the most intrusive and confrontational.
External entities conduct auditing activities to verify the accuracy of data reported to the agency.
Further, the auditors will examine the system of controls to prevent fraud, waste, and abuse.
Their tasks are to evaluate and report deviations from established procedures and policies.
Legislative and fiscal auditors have standardized sampling and review procedures that allow for
comparisons across groups. Some auditing functions (see Utah external auditing - Appendix E)
are required by policy; however, this is typically associated with fiscal accountability.
3.4.4. On-Site Monitoring - The agency uses data from IA, risk assessments, and other
control activities to conduct on-site reviews. The agency notifies each local superintendent
regarding the on-site monitoring of selected schools and the district’s central office. Standardized
interview protocols, document review checklists, and other information collection tools are
developed and assigned to each team member. The monitoring design calls for the verification
of data reported by the school and school district. On-site information is collected from data
providers, counselors, administrators, and program (including information managers) staff at the
central office.
On-site monitoring has high costs to both the agency and data collectors, mostly in time
to prepare, conduct, and respond to the findings. This control differs from “pure audit” because
the monitoring team is attempting to examine current practices with the intent to resolve poor
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 36
practices, rather than apply sanctions and consequences. Similar to the Title I monitoring
approach used in many states, the data collectors benefit by being provided with a series of
recommendations and operational details necessary to resolve the underlying problem.
For example, districts and schools should maintain documentation on all of the students
who leave the district. This documentation on who, when, and why the student left should be
reflected in the data used by the agency for graduation calculations. Inaccuracy in the exit codes
should be resolved at the source, especially when it reflects brinksmanship. On-site monitoring
costs often provide helpful information about where additional professional development efforts
should be focused.
3.4.5. Technical Manuals- These documents can provide detailed business rules, design
logic, and previous rate distributions necessary to validate task fidelity. Also, technical manuals
are one approach used to document quality assurance efforts. Technical manuals organize the
business rules, design logic, and operational programming in such a manner as to promote
transparency and increase system credibility. Business rules are often triangulated against policy
and computer languages to ensure alignment between these interactive subsystems. This
detection procedure continues to be implemented each year, with particular focus on policy-
driven changes.
Costs associated with the initial development can range from low to moderate based upon
how well the agency documents its internal procedures. Typically, the documentation resides in
numerous staff computers (including the programming code) on the agency’s servers, at worst, it
resides solely in the institutional knowledge of a few staff members.
3.4.6. Data collector Trainings - This control should focus on the data provider at the
school and the information managers within the district. Information systems are complicated by
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 37
their extensive codifications and jargon not usually used by school-level personnel. Data
providers at high schools are often inputting thousands upon thousands of entries over a given
school year, which leads to coding errors. Also, data providers may not have first hand
information about a given student and must rely on teachers and counselors. This phenomenon
increases the likelihood of errors produced through ineffective communications (“JP didn’t
dropout, he moved to Mexico or was that Mexico, Maine?”).
Training costs for data collectors should focus not only on software interface but data
quality. These costs are high for emerging systems and moderate for mature systems. Typically,
expenditures are assigned to activities focus on personnel using the interfaces correctly, rather
than on verifying the data input is accurate. Costs can be managed when teacher-led resources
are utilized in building capacity. Information management assets should provide the content and
then assign professional development experts to provide the in-services.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 38
Section 4
Conclusion
Quality assurance practices for educational agencies must define those specific actions
necessary to ensure credibility. Local and state agencies must have robust and standardized
controls that ensure non-systematic errors are not reported to the public. Graduation rates
calculated using a cohort method require high-quality inputs from information management
subsystems prior to score production. This interrelationship exists because student
characteristics, membership and cohort inputs are necessary to determine high school graduation,
completion, and dropout rates. These rates provide one indicator of the school’s effectiveness in
producing students ready to enter into the workforce or to begin additional schooling.
The QADM provides a series of indicators by which to monitor improvement in quality
assurance practices. These practices require the capacity (time, money, and people) necessary to
implement sufficient detection and control procedures. Areas that were initially determined to be
below Stage 2 are at an increased risk of unwanted variance entering into their graduation,
completer, and dropout results. These areas have the highest potential to introduce error and/or
misinformation into the results available for public consumption.
Controls must be established to mitigate risk through pre-screening data, standardizing
production procedures, conducting internal audits, and providing training. These efforts should
differ based on the maturity and capacity development actions of the agency. Communication
efforts must focus on the public’s understanding that graduation estimates are just that, estimates.
Rates determined using a cohort approach will produce higher quality information about high
schools.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 39
The movement toward reporting a cohort graduation rate is clear and progressing. With
all 50 governors on board, state education agencies are aiming to produce valid and reliable
reports. Toward this end, as state data systems are being built or refined, it is important to build
in quality assurance practices, which can take many forms, to ensure that the best possible data
are reported. Quality assurance efforts must seek to prevent misclassification from poor data and
design flaws. These tasks must be accomplished with scarce resources and limited time for
capacity development before the next production cycle. At the same time, SEA officials realize
that implementing quality assurance practices, any of the methods outlined in this paper, will
improve the quality of the data used in producing cohort graduation rates. The result, when
implemented carefully and systematically, will be a more valid and reliable report.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 40
References
Council of Chief State School Officers (2006). Validity threats: Detection and control
practices for state and local education officials. Washington, DC: Author.
Delaware Department of Education (2006). Technical and operational manual for 2006.
Dover, DE: Author.
Greene, J.P., and Winters, M.A. (2005). Public high school graduation and college-
Readiness rates. Working Paper No.8. New York: The Manhattan Institute.
Maine Department of Education (2006). Standards, assessment, and accountability
quality assurance plan. Augusta, ME: Author.
Mississippi Department of Education-Office of Research and Statistics (2007). Estimated
graduation completion and dropout counts and rates based on approved procedures for tracking
a cohort of students over 4 years. Jackson, MS: Author.
National Forum on Education Statistics. (2006). Accounting for every student: A
taxonomy for standard student exit codes (NFES 2006-804). U.S. Department of Education.
Washington, DC: National Center for Education Statistics.
National Governors Association (2005). Graduation counts: A report of the National
Governors Association task force on state high school graduation data. Washington, DC:
Author.
National Governors Association (2006). Graduation counts: Compact and task force
report-Guidance on state implementation and reporting. NGA Center for Best Practices.
Washington, DC: Author.
National Institute of Statistical Sciences and Education Statistics Services Institute
(2004). National Institute of Statistical Sciences/Education Statistics Services Institute Task
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 41
Force on Graduation, Completion, and Dropout Indicators (NCES 2005-105). U.S. Department
of Education, Washington, DC: National Center for Education Statistics.
No Child Left Behind Act of 2001, P.L. 107-110, 115 Stat. 1425 (2002); and 34 CFR Part
200 Title I-Improving the Academic Achievement of the Disadvantaged, Final Regulations
Section 200.19(a)(1).
Pyzdek, T. (2003). The Six Sigma handbook. New York, NY: McGraw-Hill.
Seastrom, M., Hoffman, L., Chapman, C. and Stillwell, R. (2007). The Averaged
Freshman Graduation Rate for Public High Schools From the Common Core of Data: School
Years 2002–03 and 2003-04 (NCES 2006–606rev). U.S. Department of Education, National
Center for Education Statistics. Washington, DC.
Seastrom, M., Chapman, C., Stillwell, R., McGrath, D., Peltola, P., Dinkes, R., and Xu,
Z. (2006).User’s Guide to Computing High School Graduation Rates, Volume 1: Review of
Current and Proposed Graduation Indicators (NCES 2006-604). U.S. Department of Education,
National Center for Education Statistics. Washington, DC: U.S. Government Printing Office.
Smith, N. J (2006). Implementing the NGA graduation rate compact: State-level issues.
Washington, DC: CCSSO.
Swanson, C.B. (2003). Keeping and losing count: Calculating graduation rates for all
students under NCLB accountability. Washington, DC: The Urban Institute, Education Policy
Center.
Swanson, C.B. (2004). Who graduates? Who doesn’t? A statistical portrait of public high
school graduation, Class of 2001. Washington, DC: The Urban Institute, Education Policy
Center.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 42
Wheeler, D.J. (2003). Making sense of data: SPC for the service sector. Knoxville, TN:
SPC Press.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 43
APPENDIX A
AYP Graduation Rate Designs across ASR Member States (NISS, 2004)
State Indicator Type
Algorithm Data Elements
AL Departure
G = High School Graduates Students receiving a standard diploma for completion of state-developed requirements for a public secondary education program in year y. Includes: recipients of Alabama High School Diploma, Alabama Occupational Diploma, and Alternate Adult High School Diploma; and summer graduates. Excludes: recipients of non-standards-based completion certificates or GED. A = Alternative High School Completers Students receiving non-diploma high school completion or exit documentation (e.g., graduation certificates). Excludes: recipients of GED or other high school equivalency credential. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition with alternative reporting calendar.)
AK Departure
G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: summer graduates. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). A = Alternative High School Completers Students completing a public secondary education program in year y without receiving a standard diploma. Includes: nonstandard diplomas, certificates (e.g., Certificates of Achievement), and GEDs. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition with alternative reporting calendar.)
AZ Cohort
G = High School Graduates Students from an entering freshman class who receive a standard diploma for completion of a public secondary education program within four years of starting ninth grade. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). E = Entering Cohort Number of students who started high school (i.e., ninth grade) in year y-3. Cohort membership (m) is defined as individuals who were enrolled for the first time in a particular grade (9) at a given point in time (y-3) within a public school system (e.g., school, district). I = Inflow to Cohort Students who joined the original cohort by transferring into the local school system at cohort grade-level. O = Outflow from Cohort Students who leave the original cohort due to transfer from the local school system or death. Excludes: dropouts.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 44
State Indicator Type
Algorithm Data Elements
CA Departure
G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: students who receive high school equivalency certificate or GED. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State adopted CCD dropout definition in 2003.)
CT Departure
G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: special education students who have until age 21 to earn a regular diploma. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)
DE Departure
G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: students who receive a GED certificate. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)
HI Cohort
G = High School Graduates Students who receive a standard diploma for completion of a public secondary education program within four years of starting ninth grade. Includes: recipients of a standard (Board of Education or Department of Education) diploma. Excludes: other program completion credentials offered in lieu of a standard diploma (e.g., IEP certificate) and equivalency credentials (e.g., GED). E = Entering Cohort Number of first-time ninth grade students in year y-3. Cohort membership (m) is defined as individuals who were enrolled for the first time in a particular grade (9) at a given point in time (y-3) within a public school system (e.g., school, district). O = Outflow from Cohort Students who leave the original cohort due to transfer from the local school system. Excludes: dropouts (as defined by CCD).
IA Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: students receiving regular diplomas from an alternative placement within the district, or who have had the requirements modified in accordance with a disability. Excludes: other program completion credentials offered in lieu of a standard diploma (e.g., certificate of attendance) D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. Includes: GED recipients. (State uses CCD dropout definition.)
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 45
State Indicator Type
Algorithm Data Elements
IL Cohort
G = High School Graduates Students who receive a standard diploma for completion of a public secondary education program within four years of starting ninth grade. Includes: recipients of a standard diploma. Excludes: students with too few credits to graduate, other program completion credentials offered in lieu of a standard diploma (e.g., IEP certificate) and equivalency credentials. E = Entering Cohort Number of first-time ninth grade students in year y-3. Cohort membership (m) is defined as individuals who were enrolled for the first time in a particular grade (9) at a given point in time (y-3) within a public school system (e.g., school, district). I = Inflow to Cohort Students who graduated in year y but were not members of the original entering cohort. Includes: students transferring into the local school system at cohort grade-level, and students who graduated in fewer or more than four years. O = Outflow from Cohort Students who leave the original cohort due to transfer from the local school system or death. Excludes: students who drop out or are expelled.
IN Other D = High School Dropouts Students who were enrolled in school during school year y or the previous summer recess and leave the educational system during that period without graduating from high school. Excludes: death, temporary absence due to suspension or a school-excused absence, and transfer to a public or nonpublic school. (State does not use CCD dropout definition.) E= Enrollment Students enrolled in grade g during year y. DR = Dropout Rate Calculated for year y by dividing number of students who drop out of grade g by number of students enrolled in that grade at the beginning of the school year.
LA Other
D = High School Dropouts Students who were enrolled in grades 9-12 at some time during school year y who were no longer in enrollment in school on October 1 of the following year. Includes: students expected to be in membership in year y+1 but have not graduated from high school or completed a state- or district-approved secondary educational program. Excludes: transfer to another public school district, private school, or state- or district-approved education program which might include a GED preparation program; temporary school-recognized absence due to suspension or illness; and death. (State uses CCD dropout definition with alternative reporting calendar.) E = High School Enrollment Cumulative student enrollment in grades 9–12 for school year y. Includes: any dropouts not included in cumulative enrollment (e.g., summer dropouts).
MA Cohort G = High School Graduates Students who took the tenth grade state assessment in year y-2 and graduated with a competency determination in year y. Excludes: students who have transferred into or out of the school system since the time of the tenth grade assessment. E = Enrollment Base Number of students enrolled in grade 10 in year y-2 (i.e., enrollment at the time of the tenth grade assessment). I = Inflow Students who joined the graduating class since the time of the tenth grade assessment by transferring into the local school system at grade-level. O = Outflow Students who leave the graduating class since the time of the tenth grade assessment by transferring out.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 46
State Indicator Type
Algorithm Data Elements
ME Departure
G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: students receiving a diploma after an approved fifth year of extended study. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. Excludes: students enrolled in an approved fifth year.
MI Cohort
G = High School Graduates Students from an entering freshman class who receive a standard diploma for completion of a public secondary education program within four years of starting ninth grade. Includes: recipients of a standard diploma. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials. E = Entering Cohort Number of students who started high school (i.e., ninth grade) in year y-3. Cohort membership (m) is defined as individuals who were enrolled for the first time in a particular grade (9) at a given point in time (y-3) within a public school system (e.g., school, district). I = Inflow to Cohort Students who joined the original cohort by transferring into the local school system at cohort grade-level. O = Outflow from Cohort Students who leave the original cohort due to transfer from the local school system or death.
MN Departure
G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)
MS Cohort G = High School Graduates Students who receive a standard diploma for completion of a public secondary education program in year y. E = Enrollment Base Number of ninth grade students in year y-3. I = Inflow to Cohort Students who joined the original cohort by transferring into the local school system at cohort grade-level. O = Outflow from Cohort Students who leave the original cohort due to transfer from the local school system. F = Failing Students Number of students at cohort grade-level failing over the four-year period. (Note: Based on review of state documents, “failing” is understood to mean students retained in grade.)
NC Other
G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). m = Graduating Class Indication of student’s expected graduating class, determined on time elapsed since taking the state’s eighth grade assessment. For the on-time graduation class in year y (i.e., m=1), students would have taken the eighth grade test in year y-4.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 47
State Indicator Type
Algorithm Data Elements
NE Departure G = High School Graduates Students receiving a standard diploma for on-time completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)
OH Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: summer graduates. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)
OK Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: summer graduates. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). A = Alternative High School Completers (GED) Students receiving GED credentials. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition with alternative reporting calendar.)
PA Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). A = Alternative High School Completers Students receiving nonstandard diplomas. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)
RI Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)
UT Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. Includes: students’ program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). (State uses CCD dropout definition.)
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 48
State Indicator Type
Algorithm Data Elements
WV Departure G = High School Graduates Students receiving a standard diploma for on-time completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 49
APPENDIX B
Graduation Rate Design Comparisons
Designer Description Unique Business Rules Limitations
National Governor’s Association (NGA) Compact Rate
The rate is an adjusted cohort graduation rate based on the number of on-time graduates in Y1 divided by first time 9th graders in Y-4 + transfers in – transfers out.
4yr Graduates- Student receives diploma in four years, excluding GED, certificates or other school-based credit. Late Completers: IEP and Immigrants assigned to later cohort.
Designed for high schools with at least grades 9-12 Standard number of years is defined as four years 4 yr vs. 5 yr rates
Louisiana’s NGA Hybrid
The rate is an adjusted cohort graduation rate based on the number of on-time graduates, early graduates, and IEP 4+graduates divided by first time 9th graders in Y-4 + transfers in – transfers out.
Establish denominator For schools, start with first-time freshmen to establish the cohort. Remove legitimate exits. Add anyone transferring into the school prior to Oct. 1, 11th-grade. Add anyone transferring from another school within the district prior to Oct. 1, 12th-grade. Remove students with IEPs that indicate they will take longer than 4 years to graduate. Add any “IEP” students that graduate or that turned 22 without graduating. Establish numerator-All those graduating with a diploma in 4 years. Any early graduates from the specific cohort. IEP graduates (4+) under age 22. District will be the same, except we add transfers into the district prior to Oct. 1, 12th-grade.
“Legitimate” leavers are those students with exit the system but cannot be directly monitored within the operational parameters of the state’s student information system. For example, expelled or students transferring to private or home school settings. State will adjust exit reason if the student is found (transferred) in another school district; conversely students coded as transferred to another district but are “no shows” are redesigned as “dropouts”.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 50
Designer Description Unique Business Rules Limitations
*Average Freshman Graduation Rate (AFGR)
The rate is an estimate of the percentage of high school students who graduate on time by dividing the number of graduates with regular diplomas by the size of the incoming freshman class 4 years earlier, expressed as a percent
Average Freshman Enrollment- Y-5 Grade 8 fall enrollment + Y-4 Grade 9 fall enrollment + Y-3 Grade 10 fall enrollment / 3 Graduating-12 grader receiving a regular diploma in Y-Total number of students graduating with a regular diploma in a single year (y)
Includes all graduates (repeaters, early graduators)-not truly on-time; Rate does not adjustments for movement in or out of the cohort. Requires grade 8 data with unclear handling rules for complex “feeding” pattern within the district. Total number of first-time 9th-grade students from three years prior to the chosen year (y-3)
Cohort Graduation Indicator (CGI)
The rate of students starting in C-4 and graduating with a regular diploma in four years + IEP five + years
9th Grade Cohort Y-4 First time 9th grades – Transfers out- GED Year and grade level a student transfers into a cohort. Year of graduation.
Underestimates the EACGI graduation indicator because it does not explicitly adjust for exclusions Requires the year a student enters the 9th grade and whether he/she enters 9th grade for the first time. A reason for a student's departure prior to graduation and documentation related to the departure.
Exclusion-Adjusted Cohort Graduation Indicator (EACGI)
The rate of students from one freshman class (first time freshman) and new entrants to that class (over the four years of high school) who graduate. Includes new members to the cohort and adjusts for documented transfers.
Annual data is collected for each student. School report/track students as they progress through high school Repeating 9th graders are differentiated from first time 9th graders. “Qualified departures” are (1) transfers to other institutions offering a state-designated diploma-granting program (2) imprisonment, and (3) death).
Requires the year a student enters the 9th grade and whether he/she enters 9th grade for the first time, year and grade level a student transfers into a cohort, and the year of graduation Schools must document reason for a student's departure prior to graduation and documentation related to the departure.
Mississippi Cohort The rate tracks a full cohort of students using the state’s student information system (MSIS) who graduate in four years. All students are included in a cohort.
Students entering as ninth graders one year and adds students entering the next higher grade in each subsequent year. Students assigned to special education self-contained grade code 56 or 58 enter the cohort based on "peer grade" (calculated using birth dates). Summer activity codes from 2004 and 2005 are included in the procedure and unknown students (predominantly students lost during the summers of 2002 and 2003) are apportioned into estimated dropouts and estimated transfers1.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 51
Designer Description Unique Business Rules Limitations
MN HSGR The rate is calculated for a particular year by counting the number of graduates divided by the number of graduates plus selected dropouts from that year and previous three years. The most recent fiscal year is referred to as Year 4 while the previous fiscal years are referred to as years 3, 2, and 1.
Establish numerator The count of students graduating in Year 4. Establish denominator Add grade 9 students dropping out in year 1-grade 10 students dropping out in year 2-grade 11 students dropping out in year 3-grade 12 students dropping out in year 4-students graduating in year 4.
Graduates from any other year (beyond the current) are not included. Only those dropouts reported in certain grades in certain years are included in the denominator with the graduates. Requires data from grades 8-12, which produce an unduplicated record set.
*Greene Graduation Indicator (GGI)
The rate is an estimate of the percentage of grade 12 students receiving a regular diploma in Year 1 divided by the adjusted grade 8 enrollment Y-4.
Adjusted 8th Grade Enrollment- Y-4 Grade 8 fall enrollment * % enrollment change Y and Y-4. Step 1-Estimate the rate of change in the high school population form the time a freshman class started to the time it was to have graduated. Step 2. This rate of change is multiplied by the estimate of first-time freshmen that takes changes due to migration into account. Step 3. Number of diplomas awarded in the graduating year is divided by the estimated number of potential on-time seniors. Calculation of GGI requires information on:
Includes all graduates (repeaters, early graduators)-not truly on-time; not adjustments for movement in or out; reduces the estimate of first-time freshman by ignoring 8th and 9th grade drop outs; assumes changes in the population over the 4 years is due to migration. Design treated cases with less than 200 9th graders as “missing”, also they cautioned against its use with highly mobile populations. Requires the number of graduates receiving a regular diploma in a specified year, the high school population (grades 9 through 12) for a specified year for three years prior, the number of 8th graders enrolled four years prior, and the 10th grade students enrolled two years prior.
*Swanson’s Cumulative Promotion Indicator (SCPI)
The index is an estimate of the probability that a student will complete within the standard four year period.
Cumulative product of the proportion of students who progress from one grade to the next at the end of the school year for grades 9, 10, and 11 multiplied by the proportion of seniors who graduate at the end of the school year.
Requires the number of students in 9th, 10th, 11th, and 12th grades in the year of analysis and the umber of students in 10th, 11th, and 12th grades in the year after the year of analysis.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 52
APPENDIX C
Selected auditing practices of selected states
Arizona Delaware Hawaii Indiana Louisiana Maine Nebraska Utah Internal auditing x Data collector verifications x x Rate assignments x x Trend analysis x Screening report templates x x Risk assessments x External audits x On-site monitoring x Technical manuals x x x x Data collector trainings x x
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 53
Appendix D
Mississippi Department of Education Report Estimated Graduation, Completion and Dropout Counts and Rates Based on Approved1
Procedures for Tracking a Cohort of Students Over 4 Years
(Final Report)
This report presents information on procedures used for tracking a cohort of students using data entered by school districts into the Mississippi Student Information System (MSIS). The target cohort comprised students entering ninth grade during the 2001/2002 school year plus students entering one grade level higher in each subsequent school year through 2004/2005. Thus, the study tracked the full cohort of students over a four-year period. The statistics presented in this report are "4-year" estimates of graduation, completion, and dropout results for the cohort. Work on this study could not begin until June 2005 -- after Month 9 (end of year) data had been received from all school districts. Since MSIS was implemented statewide in 2001/2002, that was the point at which four full years of student level data were available. The results presented in this report also include data from 2004 and 2005 "summer activity." Interpretations of data appearing in this report must acknowledge the following caveats and limitations: • The results are only as accurate as the data entered into MSIS. • The results include data from the initial implementation of MSIS. • The full cohort used in this report included repeaters. • There was no opportunity for districts to enter final disposition codes for students leaving school during the summers of 2002 and 2003. • Statistical procedures were used for apportioning students with unknown final dispositions into estimated dropouts and transfers. 1Approved through the state APA (public comment) process and State Board of Education action. Office of Research and Statistics
February 12, 2007
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 54
Introduction Information about dropouts, high school completers, and high school graduates is important to educators and the public for assessing the success of students and their educational experiences. Although the concept of a school dropout, completer, or graduate may seem simple, there are many variables that must be considered. Some states have not attempted to calculate and report one or more of the above counts or rates, and the procedures and formulas used by states has varied considerably with no standardization. Of the three rates, graduation rate is the one that has been reported by most states. Over the years, several studies have argued that graduation rates reported by states are inaccurate and are almost always too high. Some of those studies used reported enrollment to estimate state level (and, in some cases, district level) graduation rates. Although estimates based on enrollment, with or without adjustments for population changes, are not as accurate as rates based on tracking individual students over time, few states have had the data required for tracking individual students. Three sets of independent state level graduation rate estimates for Mississippi are shown in Table 1. Although the estimated rates differ and represent different student cohorts, all are considerably lower than the graduation rates reported by the state for the same years (80.5% for 2002 and 83.7% for 2004). Table 1 Independent Graduation Rate Estimates for Mississippi
Averaged Freshman Graduation Rate for the Graduating Class of 20041
Manhattan Institute Estimate Using the "Greene Method" for the Graduating Class of 20022
Urban Institute Projection Using the Cumulative Promotion Index for the Graduating Class of 20043
62.7% 60.0% 58.0% 1The averaged freshman graduation rate is based on a simple calculation using enrollment data and a count of graduates. This calculation is part of the more elaborate "Greene Method" cited below, but it does not require an adjustment for overall population change. Averaging the 8th, 9th, and 10th grade enrollments over three years helps eliminate the peak in 9th grade enrollment attributable to retention in that grade. This method approximates an ultimate graduation rate. The U.S. Department of Education adopted this procedure for reporting state level graduation rates beginning in fall 2005. See Seastrom, M., Hoffman, L., Chapman, C. and Stillwell, R. (2006) The Averaged Freshman Graduation Rate for Public High Schools From the Common Core of Data: School Years 2002-03 and 2003-04. Washington, DC. U.S. Department of Education [NCES 2006-606]. 2Greene, J. P. & Winters, M. A. (2005). Public High School Graduation and College Readiness Rates: 1991-2002 [Education Working Paper No. 8] New York: Manhattan Institute for Policy Research. This method approximates an ultimate graduation rate.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 55
3Swanson, C. B. (2005) Projections of 2003-2004 High School Graduates: Supplemental Analyses based on findings from Who Graduates? Who Doesn't? Washington, DC: The Urban Institute, Education Policy Center. This method approximates a 4-year graduation rate. State graduation rates have taken on increased significance due, in part, to the inclusion of graduation rate as a mandatory "other academic indicator" for Adequate Yearly Progress (AYP) under the No Child Left Behind Act of 2001 (NCLB). NCLB provides some information regarding which students can be counted as graduates and it indicates that the calculations are to represent graduation in a "standard" number of years. The "NGA Graduation Rate" A major push for standardization in the way states calculate and report graduation rates resulted from a year-long initiative on high school redesign that culminated during the July 2005 meeting of the National Governors Association. During (and following) that meeting all states agreed to develop and implement a standardized method of calculating and reporting graduation rates. The method conforms to recommendations developed by the NGA Task Force on State High School Graduation Data and presented in its report, Graduation Counts. The report, available on the NGA Center for Best Practices web site at www.nga.org/Center, included the following recommendations. 1. Immediately adopt, and begin taking steps to implement, a standard four-year, adjusted cohort graduation rate using the following formula: Graduation rate = [on-time graduates in year x] ÷ [(first-time entering ninth graders in year x – 4) + (transfers in) – (transfers out)] 2. Build the state’s data system and capacity to ensure that the system can collect, analyze, and report the adopted indicators and other important information. 3. Adopt additional, complementary indicators to provide richer context and understanding about outcomes for students and how well the system is serving them, including five- and six-year cohort graduation rates; a college-ready graduation rate; a dropout rate; completion rates for those earning alternative completion credentials from the state or a GED; in-grade retention rates; and percentages of students who have not graduated but are still in school or who have completed course requirements but failed a state exam required for graduation. At the time the above report was released, the Mississippi Department of Education (MDE) had just begun working on procedures for tracking a student cohort in MSIS for purposes of calculating accurate dropout, completion, and graduation rates. MSIS satisfies the second NGA recommendation (a comprehensive statewide student level database) and contained four years of data by the end of the 2004/2005 school year. Although summer activity for 2005 would not be available until early spring 2006, preliminary estimates of dropout, completion, and graduation rates were compiled by the Office of Research and Statistics (ORS) and presented to the State Board of Education at its meeting in August 2005.The estimated rates calculated for the "true cohort" of students entering 9th grade at the beginning of the 2001/2002 school year were: Dropouts – 26.0%, Graduates (4-Year) – 60.6%, Graduates (ultimately) – 63.2%.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 56
The above graduation rate estimates agreed closely with the independent estimates presented in Table 1. Additional work by ORS over the next 12 months would include the development of procedures for constructing a "full cohort" of students, incorporation of summer activity, and procedures for accounting for students whose final disposition (transfer, dropout, completion) was unknown. CCSSO Technical Panel The Council of Chief State School Officers gathered representatives from several states and organizations, the National Governors Association Center for Best Practices, and the Data Quality Campaign in June 2006 to evaluate and discuss implementation issues related to the NGA Graduation Rate. The goal of the technical panel was to identify standard implementation practices that could be adopted by all states to make the subsequent rates as comparable as possible. In its recently released report, Implementing the NGA Graduation Rate Compact: State-level Issues (available on the CCSSO web site at www.ccsso.org), the Technical Panel stated, "While initially the formula appears very straightforward, there are so many differences among state data systems as well as in state-specific rules for accounting for and providing services to students with special needs that cross-state comparability is not an easy feat with this one definition. This graduation rate is applied to students who receive a standard diploma, not a certificate of completion or attendance or a General Educational Development (GED) certificate." In addition to addressing data collection issues, the report discussed definitions and procedural issues such as cohort length, cohort definition, and special populations (e.g., students with disabilities and limited English proficient students). Working with the NGA compact guidelines, the CCSSO Technical Panel made specific recommendations for dealing with various data issues. The recommendations included the following (cited verbatim). • Because there may only be a small portion of the student body who is allowed additional time to graduate in their IEPs, states may choose to not include them in the NGA graduation rate cohort, and include them in a five- or six-year graduation rate… • Calculate the NGA rate for cross-state comparability, but consider calculating at least a five-year rate for students who receive a regular or advanced diploma within five years, without receiving the fifth year as a result of a special dispensation from the school, in addition in order to provide a more complete graduation picture for the state. • For the purposes of the NGA graduation rate, the panel recommends defining first time 9th graders as any student who was enrolled at least one day in grade 9 in Year X. • The panel recommends that the count of transfers in includes every student who enters the cohort on grade-level at any point during the four year period and does not exclude students who arrive late in the 12th grade (or any grade).
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 57
• Define, document, and implement a detailed student-level Exit Code data collection process to account for students who leave the public school during or between school years. • Establish a process by which the SEA reviews statistical trends of exit data within and across school years to identify potentially erroneous data. Establish a detailed review and validation process for samples of district, school, and/or student data. Establish clearly defined consequences for schools and districts that do not maintain clear and accurate documentation and validation processes that meet the states guidelines and for submitting erroneous data to the state. Clearly communicate each of the processes and consequences to districts and schools. • Students who ‘vanish’ (i.e., cannot be found in another location, no documentation exists for where they went, etc.) should be counted as dropouts, not as transfers out. • 4th-year summer graduates should be counted as graduates in the NGA rate. • Create a way to collect student-level data that identifies which students are allowed additional time to graduate by their IEPs. Calculate the NGA rate without these students the year after the end of the cohort and then again the following year with these students included. Be transparent in reporting the impact these students had on the rates and why the rate for a given year was recalculated and republished. • Follow the same general guidelines used for students receiving special education services [to deal with issues involving students with limited English proficiency]. • Follow existing state policies and practices regarding tracking and accounting for incarcerated students, but be very clear in how those students are included in the calculation of the NGA graduation rate. • Students retained in grades 9-12 remain in the cohort to which they were originally attributed. In its conclusion, the report states, "The NGA graduation rate is not intended to be used as an accountability tool by states or the U.S. Department of Education. The purpose of the NGA graduation rate is to provide governors and other policymakers with a standard definition based on student-level longitudinal and high quality data that enables comparability across states. Having a standard definition across states helps policymakers across the country communicate with and learn from each other as future research and policies are designed and undertaken. The Graduation/Completion/Dropout (GCD) Definition Committee In December 2005, the State Superintendent designated a committee to continue work on procedures for tracking cohorts of students in MSIS and calculating accurate graduation, completion, and dropout counts and rates. The committee met five times between December
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 58
2005 and October 2006 to discuss issues and review results based on data produced by the Office of Management Information Systems (MIS) and the Office of Research and Statistics. Updates of the state level results were presented to the Mississippi State Board of Education during July 2006 and October 2006. The final state level results are presented later in this paper. Although the CCSSO Technical Panel Report was only released in October 2006, the procedures developed by the GCD Definition Committee and used for estimating the state and district level counts and rates are in close agreement with the Technical Panel's recommendations. For example, the tracked cohort comprises students entering the ninth grade at the beginning of school year 2001/2002 ("true cohort" students) plus other ninth grade students entering during that school year plus students entering grade 10 in 2002/2003, students entering grade 11 in 2003/2004, and students entering grade 12 in 2004/2005. The full cohort also includes self-contained special education students who were added to the cohort based on "peer grade" (calculated using the student's age). All students remained in the identified cohort even if they were retained in a grade (9 through 12). Student transfers and deaths, as well as dropouts, were identified using school year and "summer activity" codes in MSIS. The system was designed to track the full cohort assessing the final disposition of each student at selected points (at the end of 4 years, the end of 5 years, etc.). Certain students with disabilities are counted differently in rates representing different time spans. Tracking a Cohort of Students in MSIS: The Big Picture As stated earlier in this report, preliminary tracking of the "true cohort" (students in ninth grade at the beginning of school year 2001/2002) produced an estimated 4-year dropout rate of 26.0%, an estimated 4-year graduation rate of 60.6%, and an estimated ultimate graduation rate of 63.2%. After much work developing procedures for constructing and tracking a "full cohort" and determining the final disposition for each student, the updated 4-year estimates were very similar. The steps included in the data analysis and calculations, and the intermediate results, are shown in Figure 1 (Computer Output – Step by Step). Establishing the Full Cohort Beginning with 38,833 students who were in grade 9 at the beginning of 2001/2002, 3,600 incoming ninth graders were added that year followed by 4,826 tenth graders in 2002/2003, 2,668 eleventh graders in 2003/2004, and 1,464 twelfth graders in 2004/2005. That resulted in a full cohort comprising 51,391 students. The total cohort included 2,199 self-contained special education students who had been added to the cohort based on "peer grade." Within the cohort, 7,023 students were students with disabilities (eligible under IDEA) sometime within the 4-year tracking period. Identifying Appropriate Denominators Using the transfer/death codes in MSIS (for each school year and for "summer activity" in 2004 and 2005), 7,742 students were identified as "coded" transfers/deaths. There were 4,648 students whose final disposition was unknown. Most of those students probably left (transferred, died, or
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 59
dropped out) during the summer in 2002 or 2003 – however, there was no way for districts to code the disposition for those students. Therefore, a procedure was developed for estimating the number of unknown disposition students who were probably transfers/deaths and the number who were probably dropouts. Using summer activity codes entered by districts for 2004 and 2005, it was determined that 35.0% of the students had been coded as transfers/deaths and 58.5% had been coded as dropouts. Those percentages were used to apportion the unknown disposition students into estimated transfer/death and dropout counts. Applying 35.0% to the 4,648 unknowns resulted in an estimate of 1,625 additional transfers and deaths. Thus, the total estimated transfer/death count was 9,367. That count was subtracted from the total cohort N-Count to yield a dropout denominator – 42,024. As explained later, a slightly different denominator must be used for calculating completion and graduation statistics. Estimated Dropout Rate Calculation Using the dropout codes in MSIS (for each school year and for "summer activity" in 2004 and 2005), 8,306 students were identified as "coded" dropouts. Applying the 58.5% summer activity dropout percentage value to the 4,648 unknowns resulted in an estimate of 2,718 additional dropouts. Thus, the total estimated dropout count was 11,169. Dividing that value by the dropout denominator (42,024) resulted in a 4-year dropout rate estimate of 26.6%.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 60
Figure 1. Computer Output – Step by Step
COHORT STATISTICS FOR SY0102G09 -- OVER 4 YEARS NOTE1: APPORTIONING OF UNKNOWNS INTO DROPOUT AND TRANSFER/DEATH ESTIMATES WAS BASED ON DISTRICT SUMMER ACTIVITY CODING FOR SUMMER 2004 AND 2005. NUMBER OF STUDENTS WITH UNKNOWN FINAL DISPOSITION = 4648 NOTE2: ESTIMATED DROPOUT RATE WAS APPLIED TO STUDENTS STILL IN SCHOOL TO GET AN ESTIMATE OF POSSIBLE COMPLETERS BEYOND THE 4-YEAR PERIOD. NOTE3: VALUES MARKED WITH AN ASTERISK (*) ARE SUBTOTAL VALUES THAT ARE ALREADY INCLUDED IN ANOTHER LISTED VALUE. ------------------------------------------------------------------------------ N-COUNT COHORT 0 ( 9TH GRADERS MONTH 1 2001/2002) 38833 N-COUNT COHORT 1 ( 9TH GRADERS ADDED IN 2001/2002) 3600 N-COUNT COHORT 2 (10TH GRADERS ADDED IN 2002/2003) 4826 N-COUNT COHORT 3 (11TH GRADERS ADDED IN 2003/2004) 2668 N-COUNT COHORT 4 (12TH GRADERS ADDED IN 2004/2005) 1464 N-COUNT TOTAL (TRUE COHORT + ALL ADDED STUDENTS) 51391 ADDED GRADE 56/58 (BASED ON STUDENT AGE/PEER GRADE) 2199* SPED STUDENTS IDENTIFIED FROM 2001/2002 THROUGH 2004/2005 7023* ------------------------------------------------------------------------------ LOST T3 (MAY REPRESENT MSMS, MSA, MSB, MSD, ETC.) 1068 TRANSFER T4 ( TRANSFER TO NON-PUBLIC SCHOOL) 2893 TRANSFER T5 ( TRANSFER TO SCHOOL OUT OF STATE) 2521 TRANSFER T7 ( TRANSFER TO APPROVED GED PROGRAM) 138 TRANSFER T8 ( TRANSFER TO HOME SCHOOL) 99 TRANSFER ST4 (SUMMER TRANSFER TO NON-PUBLIC SCHOOL) 217 TRANSFER ST5 (SUMMER TRANSFER TO SCHOOL OUT OF STATE) 541 TRANSFER ST7 (SUMMER TRANSFER TO APPROVED GED PROGRAM) 118 TRANSFER ST8 (SUMMER TRANSFER TO HOME SCHOOL) 70 DEATH Z1 (DEATH) 65 DEATH SZ1 (SUMMER DEATH) 12 TRANSFER/DEATH PERCENTAGE USED FOR ESTIMATE (SEE NOTE 1) 35.0% ESTIMATED TRANSFER/DEATH FOR UNKNOWN STUDENTS (SEE NOTE 1) 1625 TOTAL TRANSFER/DEATH (THIS IS AN ESTIMATE) 9367 DROPOUT DENOMINATOR (ALL UNKNOWNS CONSIDERED TRANSFERS) 39306 DROPOUT DENOMINATOR (TOTAL MINUS TRANSFER/DEATH ESTIMATE) 42024 DROPOUT DENOMINATOR (ALL UNKNOWNS CONSIDERED DROPOUTS) 43649 CODED DROPOUTS (CODED DURING THE FOUR SCHOOL YEARS) 6703 CODED DROPOUTS (CODED IN SUMMER ACTIVITY 2004 & 2005) 1603 CODED DROPOUTS WITH REASON 12 (XFER TO GED OR NON-EDUC) 2388* DROPOUT PERCENTAGE USED FOR ESTIMATE (SEE NOTE 1) 58.5% ESTIMATED DROPOUTS FOR UNKNOWN STUDENTS (SEE NOTE 1) 2718 LOST T1 (DROPOUT: NO ENTRY IN ANOTHER GRADE IN SCHOOL) 6 LOST T2 (DROPOUT: NO ENTRY IN ANOTHER SCHOOL IN THIS DISTRICT) 139 ALL DROPOUTS OVER THE FOUR YEARS (THIS IS AN ESTIMATE) 11169 ESTIMATED DROPOUT RATE (LOW: UNKNOWNS CONSIDERED TRANSFERS) 21.5% (- 5.1%) ESTIMATED DROPOUT RATE (CODED PLUS ASSUMED-SEE NOTE 1) 26.6% ESTIMATED DROPOUT RATE (HIGH: UNKNOWNS CONSIDERED DROPOUTS) 29.3% (+ 2.7%)
Continued on Next Page
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 61
Figure 1. Computer Output – Step by Step (continued)
UNKNOWNS THAT WERE NOT APPORTIONED 305 NOT APPORTIONED (PERCENTAGE OF THE DROPOUT DENOMINATOR) 0.7% ------------------------------------------------------------------------------ GRADE 56/58 STILL AROUND AT END OF 2004/2005 827 ANY GRADE LEP STILL AROUND AT END OF 2004/2005 9 COMPL/GRAD DENOM (ALL UNKNOWNS CONSIDERED TRANSFERS) 38470 COMPL/GRAD DENOM (TOTAL MINUS TRANSFER/DEATH/SA56/SA58) 41188 COMPL/GRAD DENOM (ALL UNKNOWNS CONSIDERED DROPOUTS) 42813 COMPLETERS (CODED COMPLETERS OVER THE 4 YEARS) 27589 COMPLETED WITH A TRADITIONAL DIPLOMA (COUNTS AS A GRADUATE) 25057* COMPLETED WITH AN OCCUPATIONAL DIPLOMA (CHANGE:NOT A GRADUATE) 128* COMPLETED ALL REQUIREMENTS EXCEPT FOR SATP PASSING SCORE 180* COMPLETED WITH A CERTIFICATE OF ATTENDANCE 1481* COMPLETED WITH A DISTRICT PROGRAM GED 743* ESTIMATED COMPL RATE (LOW: UNKNOWNS CONSIDERED DROPOUTS) 64.4% (- 2.6%) ESTIMATED 4-YEAR COMPLETION RATE (WITH APPROTIONED UNKNOWNS) 67.0% ESTIMATED COMPL RATE (HIGH: UNKNOWNS CONSIDERED TRANSFERS) 71.7% (+ 4.7%) GRADUATES (CODED DIPLOMA [TRAD] RECIPIENTS OVER THE 4 YEARS) 25057 ESTIMATED 4-YR GRAD RATE (LOW: UNKNOWNS CONSIDERED DROPOUTS) 58.5% (- 2.3%) ESTIMATED 4-YEAR GRADUATION RATE (WITH APPORTIONED UNKNOWNS) 60.8% ESTIMATED 4-YR GRAD RATE (HIGH: UNKNOWNS CONSIDERED TRANSFERS) 65.1% (+ 4.3%) ESTIMATED 4-YR GRADUATION RATE (INCLUDING GED RECIPIENTS) 62.6% ------------------------------------------------------------------------------ STILL ENROLLED AT THE END OF SCHOOL YEAR 2004/2005 2961 STILL ENROLLED IN GRADE 12 (AND HAS NOT COMPLETED) 467* STILL ENROLLED IN GRADE 11 1030* STILL ENROLLED IN GRADE 10 422* STILL ENROLLED IN GRADE 9 93* STILL ENROLLED IN GRADE 8 (PROBABLE MSIS DATA ERROR) 2* STILL ENROLLED IN GRADE 7 (PROBABLE MSIS DATA ERROR) 1* STILL ENROLLED IN GRADE 56 (PROBABLE MSIS DATA ERROR) 13* STILL ENROLLED IN GRADE 58 (NOTE: 56/58 NOT IN DENOMC) 814* STILL ENROLLED IN GRADE 78 62* STILL ENROLLED ASSUMPTION (UNKNOWN WITH SA CODE IS1) 28* STILL ENROLLED ASSUMPTION (UNKNOWN WITH SA CODE IS2) 18* STILL ENROLLED ASSUMPTION (UNKNOWN WITH SA CODE IS3) 11* STILL ENROLLED (AS A % OF THE COMPLETION DENOMINATOR) 7.2% POSSIBLE FUTURE COMPLETERS (SEE NOTE 2) 5.3%
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 62
Estimated Completion Rate Calculation The number of students actually coded by districts as school completers (traditional graduates, occupational diploma graduates, district GED program completers, special education certificate of attendance recipients, and students completing all requirements for graduation except for a passing score on one of the Subject Area tests) was used as the numerator for calculating the 4-year completion rate. Using the special grade code (58) for self-contained special education students, 827 students with that code who were still enrolled in school at the end of 2004/2005 (plus 9 LEP students who were still in school) were subtracted from the dropout denominator yielding a completion/graduation denominator – 41,188. Dividing 27,589 coded completers by the completion/graduation denominator (41,188) produced a 4-year completion rate estimate of 67.0%. That rate is an estimate because the denominator includes an estimated count of transfers/deaths. Estimated Graduation Rate Calculation The number of students actually coded by districts as high school graduates (traditional graduates only) was used as the numerator for calculating the 4-year graduation rate. Dividing 25,057 coded graduates by the completion/graduation denominator (41,188) produced a 4-year graduation rate estimate of 60.8%. That rate is an estimate because the denominator includes an estimated count of transfers/deaths. Future Completers and Rates Beyond Four Years Although the NGA graduation rate is explicitly the rate for students completing high school in four years, both the NGA Task Force on State High School Graduation Data and the CCSSO Technical Panel acknowledged that such a rate provides only a partial picture regarding high school completion. Until data are available for individual students with disabilities concerning the number of years their IEP team determines is needed for graduation (or high school completion), it will not be possible to accurately adjust the completion/graduation denominator. For the 4-year completion and graduation rate estimates presented in this report, students with disabilities instructed in a self-contained secondary environment and still enrolled at the end of four years were not included in the completion/graduation denominator when calculating the estimated 4-year rates. A valid assumption is that those students would not be expected to graduate from high school within four years. In fact, many of those students' IEPs state that the student will complete high school with a certificate of attendance rather than a regular or occupational diploma. In addition to self-contained students with disabilities who would not be expected to complete high school within four years, there are other students (disabled and non-disabled) who were retained in some grade 9-12 and were still in school at the end of 2004/2005. Although those students are included in the denominator and are not counted as completers or graduates in the estimated 4-year completion and graduation rates, most will eventually complete high school. Of the students in the total cohort, 2,961 (7.2% of the students in the completion/graduation denominator) were still enrolled in school at the end of 2004/2005. Applying the 4-year dropout
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 63
estimate (26.6%) to the students who were still enrolled resulted in an estimate of 2,173 students who will probably complete high school in future years.
The 2,173 additional students represent 5.3% of the completion/graduation denominator. Adding 5.3% to the 4-year completion estimate yields an estimated ultimate completion rate of about 72%.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 64
State Level 4-Year Dropout, Completion and Graduation Rate Estimates Once the procedure for tracking a full cohort of students in MSIS was approved, it was run separately for certain student subgroups to produce disaggregated estimates. Table 2 presents estimated 4-year dropout, completion, and graduation counts and rates by gender and ethnicity. Table 2 Disaggregated 4-Year Dropout, Completion, and Graduation Data
Group
Total Cohort N-Count
Dropout Denominator (Transfers and Deaths1
Subtracted)
Estimated Dropouts1
And Estimated 4 Year Dropout Rate2
Completion/ Graduation Denominator (Transfers, Deaths1 and Grade 58 Subtracted)
Completers and Estimated 4-Year Completion Rate
Graduates and Estimated 4-Year Graduation Rate3
Possible Future Completers4
All Students 51,391 42,024 11,169 26.6% 41,188 27,589
67.0% 25,057 60.8%
5.3%
Female 23,895 19,803 4,174 21.1% 19,524 14,427
73.9% 13,529 69.3
4.2%
Male 27,496 22,226 7,002 31.5% 21,669 13,162
60.7% 11,528 53.2%
6.1%
Asian 438 311 43 13.8% 308 255
82.8% 245 79.5%
3.1%
Black 26,094 22,437 6,748 30.1% 21,897 13,389
61.1% 11,839 54.1%
6.6%
Hispanic 508 286 74 25.9% 273 189
69.2% 174 63.7%
6.0%
Native American 107 65 26
40.0% 64 33 51.6%
30 46.9%
5.6%
White 24,244 18,838 4,227 22.4% 18,559 13,723
73.9% 12,769 68.8%
3.6%
Black Female 11,971 10,514 2,484 23.6% 10,343 7,166
69.3% 6,640 64.2%
5.5%
White Female 11,398 8,929 1,619 18.1% 8,828 7,012
79.4% 6,651 75.3%
2.7%
Black Male 14,123 11,923 4,266 35.8% 11,554 6,223
53.9% 5,199 45.0%
7.3%
White Male 12,846 9,913 2,613 26.4% 9,735 6,711
68.9% 6,118 62.8%
4.4%
Note: This table represents the official final run on 02/11/2007 and supersedes all earlier versions.
1Based on actual statewide 2004 and 2005 summer activity coding, 58.5% of unknown students were classified as dropouts and 35.0% were classified as transfers/deaths. 2Includes all coded school year and summer activity dropouts plus "lost" T1 and T2 transfers. This represents a 4-year "9-12" dropout rate. The customary "7-12" cohort dropout rate would be higher. 3Graduates include only traditional diploma recipients. Occupational diploma recipients, district GED recipients, special education certificate of attendance recipients, and students who completed all requirements except for a passing score on one or more tests required for graduation are completers, but not graduates. Note: Occupational diploma classification was changed on the 02/11/2007 run.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 65
4Possible future completion percentage was calculated by applying the estimated statewide dropout rate to students who were still enrolled at the end of 2004/2005. Add the percentage in this column to estimate the ultimate completion rate; the estimated ultimate graduation rate will be somewhat lower.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 66
A Procedure for Tracking a Full Cohort in MSIS: The Details This section of the paper provides details concerning the procedure developed for tracking a full cohort of students in MSIS and calculating estimated dropout, completion, and graduation rates for the full cohort. The objectives were to ensure that all students would be included in an appropriate cohort (i.e., no students would be systematically left out) and to use data available in MSIS to calculate accurate counts and rates for dropouts, completers, and graduates. Since MSIS was implemented statewide at the beginning of the 2001/2002 school year, the first four-year full cohort could be tracked using data from the end of school year 2004/2005 updated with the results of the 2005 summer activity procedure. All the required data were available in April 2006. Using the most appropriate analysis techniques, there are still certain errors that will be present in the calculated counts and rates. Those include coding errors in the data transmitted to MSIS from the district student administrative packages (SAPs), a small amount of data inconsistency (generally in the first school year) due to edit checks that were implemented later, and students whose final disposition is unknown because they left the system during summer 2002 or summer 2003 (before the summer activity procedures were implemented). Logic for implementing a full cohort tracking system was approved and data files were built using the data in MSIS. The steps used for tracking the cohort and analyzing the data follow.
1. All students who entered MSIS as ninth graders during school year 2001/2002 were identified and their data written to a data file. For each student, a variety of data variables were extracted from MSIS, including the last know disposition for students who were not still enrolled somewhere in Mississippi at the end of 2004/2005. [N=41,160]
2. All students who entered MSIS as tenth graders during school year 2002/2003, and were not already in the data file, were identified and added to the file. [N=4,384]
3. All students who entered MSIS as eleventh graders during school year 2003/2004, and were not already in the data file, were identified and added to the file. [N=2,344]
4. All students who entered MSIS as twelfth graders during school year 2004/2005, and were not already in the data file, were identified and added to the file. [N=1304]
5. All students coded grade 56 or 58 (self-contained special education) who were the age of typical ninth graders during school year 2001/2002 were identified and their data written to a separate data file. [N=1,310]
6. All students coded grade 56 or 58 (self-contained special education) who were the age of typical tenth graders during school year 2002/2003, and were not already in the data file, were added to the file. [N=452]
7. All students coded grade 56 or 58 (self-contained special education) who were the age of typical eleventh graders during school year 2003/2004, and were not already in the data file, were added to the file. [N=331]
8. All students coded grade 56 or 58 (self-contained special education) who were the age of typical twelfth graders during school year 2004/2005, and were not already in the data file, were added to the file. N=[164]
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 67
9. Student records in the regular grade file and the grade 56/58 file were matched by MSIS ID to identify any duplicates. Of the 2,257 grade 56/58 students, 58 students had records in both files. An examination of the duplicate data records revealed that the only difference was in the cohort flag (year of entry into the cohort). Therefore, a procedure was run to eliminate one of the duplicate records and merge the regular grade and grade 56/58 data files. Unduplicated records comprised the full cohort. [N=51,391]
10. A tentative solution was used for student data records that contained both a completion code and a transfer code. This problem reflected data from the earliest year, prior to the addition of certain edit checks in MSIS. The solution was to blank out the transfer code for any student with a completion code (T, O, G, OD).
11. The cohort flag, transfer codes, and completion codes in the data file were used to set unique binary (0,1) variables that could be used for aggregating the student data at the school, district, and state levels.
12. The full cohort data file was matched by MSIS ID to the Month 1 2001/2002 enrollment file to determine which students coded as ninth graders in 2001/2002 were "true beginning" cohort students rather than students that had entered later in the school year. Of the 41,160 students, 38,833 were true cohort students and 2,327 had been added.
13. The full cohort data file was matched to the Month 9 2004/2005 enrollment file to determine the status for students who had no transfer code and no completion code. There were 145 "lost" T1 and T2 transfer students and 2961 students enrolled in some grade.
14. For students in the full cohort data file whose disposition was still unknown after the above steps, summer activity codes were applied. Of the 2,741 students with summer activity codes from 2004 or 2005, 6.6% had completed all requirements except for a passing score on one or more tests needed for graduation, 58.5% had been coded as dropouts, and 35.0% had been coded as transfers or deaths.
15. Ultimately, there were 4,648 students in the full cohort data file whose final disposition was unknown. Since they were probably students who were lost during summer 2002 and summer 2003 (prior to the implementation of the summer activity process), it was decided to apply apportioning constants to the aggregate counts based on the actual percentage of students coded as summer dropouts and transfers during 2004 and 2005.
16. All the full cohort data records were summarized (aggregated) at the state level based on tentative logic outlined in the steps that follow.
17. Binary values (0 or 1) were accumulated across all student records for each of the following variables. Total Full Cohort N-Count, Cohort 0 (true cohort), Cohort 1, Cohort 2, Cohort 3, Cohort 4, T1, T2, T3, T4, T5, T7, T8, Z1, IS1, IS2, IS3, ST4, ST5, ST7, ST8, SZ1, Dropout, Dropsumm, Diploma, Trad, Occu, GED, Cert, ABT, Stillenrl, and Unknown.
18. The unknown student aggregate count was multiplied by 0.585 (58.5%) to yield the statewide number estimated to have been summer dropouts (SD1 through SD21).
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 68
19. The unknown student aggregate count was multiplied by 0.350 (35.0%) to yield the statewide number estimated to have been summer transfers/deaths (ST4, ST5, ST7, ST8, SZ1).
20. All transfer and death counts (coded and estimated) were accumulated and subtracted from the total full cohort N-count to yield the denominator for calculating an estimated 4-year dropout rate. [Dropout Denominator Value=42,024]
21. Accumulated counts for students coded as dropouts during the school year, those coded as dropouts during 2004 and 2005 summer activity, those estimated as dropouts during summer 2002 and 2003, and "lost" T1 and T2 transfers to yield an estimate of dropouts over the 4-year period. [Value=11,169]
22. Divided the dropouts count by the denominator to yield the estimated 4-year dropout rate. [Value=26.6%]
23. Subtracted from the dropout denominator value the count of certain “still enrolled” students who would not be expected to complete high school within four years. Since the IEP for students with disabilities under IDEA do not currently contain a specific IEP committee recommendation regarding the number of years those students need to successfully complete high school, a “proxy” measure was used. Cohort students who were still enrolled in self contained special education environments (i.e. MSIS grade code “58”) at the end of 2004/2005 were assumed to be students for whom an IEP committee would recommend extended time for school completion [Value = 827]. In addition, 9 LEP students who were still enrolled at the end of 2004/2005 were considered students who would be expected to take longer than four years to complete high school. Subtracting those 836 students from the dropout denominator (42024) yielded the denominator for calculating estimated 4-year completion and graduation rates. [Completion/Graduation Denominator Value=41,188]
24. Accumulated counts for traditional diploma recipients, occupational diploma recipients, GED recipients, SPED certificate of completion recipients, and students lacking only a passing score on a test required for graduation to yield an estimate of school completers over the 4-year period. [Value=27,589].
25. Divided the completers count by the completion/graduation denominator to yield the estimated 4-year completion rate. [Value=67.0%]
26. Divided the traditional diploma count [Value=25,057] by the completion/graduation denominator to yield the estimated 4-year graduation rate. [Value=60.8%]
27. Applied the estimated dropout rate to the count of still-enrolled students to estimate the percentage of students in the denominator who might be expected to complete school in future years. [Value=5.3%] Note: Some of the students who were still enrolled were self-contained special education students. Therefore, the possible completion rate can be added to the estimated 4-year completion rate to estimate an ultimate completion rate [about 72%], but the estimated ultimate graduation rate would be somewhat lower.
28. Calculated disaggregated dropout, completion, and graduation rates by going back to step #16 and running the analyses separately by gender (Female, Male), ethnicity (Asian, Black, Hispanic, Native American, White) and gender/ethnicity (Black Female, White Female, Black Male, White Male).
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 69
Procedure for Calculating District Level Estimates
29. District level cohort statistics were calculated using the procedures described above. Beginning at Step #16, the full cohort data were aggregated to the district level based on the last known district in which the student was enrolled.
30. In step #18, the percentage of summer activity records coded as dropouts by the last known district during the 2004 and 2005 summer activity process was applied to the unknown student aggregate count to yield an estimated number of dropouts.
31. In step #19, the percentage of summer activity records coded as transfers/deaths by the last known district during the 2004 and 2005 summer activity process was applied to the unknown student aggregate count to yield an estimated number of transfers/deaths.
Issues Related to Calculating School Level Estimates (No expected completion time available)
The calculation of school level dropout, completion, and graduation rates will require special school-to-school linking. For example, some students begin in a school containing a 9th grade, but end in a different school containing a 12th grade. Students whose final disposition (transfer, dropout, death) occurs in 9th grade would have a "last known school" differing from the rest of the cohort. Procedures will have to be developed for handling such situations. Differences in Graduation Rate Calculations The large variation in graduation rates produced by the traditional Mississippi formula and the procedures presented in this paper is due, primarily, to differences in the way that the data are managed and the calculations are performed. The differences are outlined in the following table.
Traditional Formula Cohort Tracking Procedure
1. Students included in a cohort Only students who enter grade 9 are entered into a cohort. Therefore, self-contained students with grade code "58" are never included in a cohort and never appear in the denominator for calculating the graduation rate. That reduces the size of the denominator and increases the graduation rate.
All students are eventually included in a cohort. Grade 58 students are included in the denominator for calculating dropout rates, however, they are handled differently for calculating completion and graduation rates for different time spans (4 years, 5 years, 6 years).
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 70
Traditional Formula Cohort Tracking Procedure
2. Students leaving during the summer
A procedure was implemented beginning in 2004 to account for students who were present at the end of one school year and did not show up at the beginning of the next school year. However, all graduation rate calculations that included students leaving during earlier summers had those students subtracted from the cohort. That reduced the size of the denominator and resulted in an increased graduation rate.
Although the procedures used in this study included the same students that left during the summers of 2002 and 2003, those students weren't excluded from the cohort. Instead, a procedure was implemented for apportioning the students into estimated transfers and estimated dropouts based on summer activity coding for 2004 and 2005.
3. Students completing school with a GED, an occupational diploma, or a certificate of attendance.
While not counted as graduates, students completing high school with a GED, an occupational diploma, or a certificate of attendance are subtracted from the denominator. That is virtually the same as actually counting them as graduates and it increases the graduation rate.
Students completing high school in any way are counted as school completers. However, only those earning a traditional diploma are counted as graduates. The same denominator is used for calculating the completion rate and the graduation rate.
4. Students retained in a high school grade.
Students retained in a grade 9-12 are subtracted from their original cohort count and added to the cohort count for the next later graduating class. Although this does not change the ultimate graduation rate, it makes it impossible to calculate a true "4-year" graduation rate.
Students remain in the same cohort. Students taking longer than four years to complete will not count in the 4-year graduation rate, but will count in the rate for their graduation year (e.g., the 5-year or 6-year rate). Note: Certain students with disabilities and certain LEP students are excluded from the denominator when calculating certain rates (e.g., the 4-year graduation rate).
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 71
APPENDIX E
Graduation Rate Proxies: Utah and Minnesota
Graduation, completers, dropout proxies: Utah
This definition implies a cohort rate. Utah is exercising the option under 200.19(a)(1)(b)
to adjust this definition slightly in order to ensure statewide comparability and reduce error in
measurement by restricting the cohort to grades ten through twelve, since Utah high schools,
which may implement any of three different grade spans, have only these three grades in
common; in fact, nearly half have only these three grades -- Grades 7-12 (21% of high schools),
Grades 9-12 (33%), and Grades 10-12 (46%).
Specifically, the cohort graduation rate is operationalized by Utah, according to the
recommendation of the NCES (U.S. Dept. of Ed., August 2002, p. 3), to simulate the movement
of a class through high school: The number of students who graduated from 12th
grade in the
current year divided by the sum of: (1) these same graduates, and (2) the number of students who
dropped out of 12th
grade in the current year, (3) 11th
grade in the prior year, and (4) 10th
grade in
the year before that.
In order to continue applying official NCES definitions (U.S. Dept. of Education, January
2003, pp. 25, 79-81) -- in distinguishing "graduates" from "other completers", and "dropouts"
from "transfers" [see Note below] -- which have already been incorporated into Utah State Board
of Education rule (R277-419), Utah also lags the rate by one year; thus, the 2003 report includes
the rate for the 2002 cohort.
Regular diploma graduates may include students with disabilities who can be retained as
"seniors" until the age of 22. As long as such students are retained, their cohort year is adjusted,
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 72
so that their completion status is included in the calculation of the graduation rate for the
graduating class of the year in which it is finally determined.
Since graduation counts are derived from student level detail, they can be disaggregated
by all of the programmatic and demographic categories identified in NCLB legislation. Dropout
counts, however, are only available in disaggregated form by gender and ethnicity, as required
by the NCES in its Common Core of Data (CCD) surveys, so graduation rates cannot be
calculated for all subgroups.
To address the discrepancy between NCES/CCD and NCLB/AYP in disaggregation
requirements, Utah will begin collecting dropout counts at the student level during the 2004-05
school year, when the Class of 2007 is in 10th grade.
Graduation, completers, dropout proxies: Minnesota
AYP Calculation: Graduation Measurement
The Graduation measurement is one of the Secondary Indicators. The Graduation
computation is based on MARSS enrollment data reported over a five year period; End of Year
data from the previous four years and Fall data from the current year. The graduation measure is
computed for all disaggregated groups, but is only used as an AYP Secondary Indicator measure
for the ‘All’ group where required. The disaggregated group AYP marks are only used to
determine Safe Harbor status when computing Proficiency.
Target
80% (0.8000) or 0.1% (0.0010) improvement over the prior year.
Minimum Cell Size for Measurement
40
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 73
Student record selection
Generally, students in grades 8-12 within a school are evaluated to compute the
Graduation measurement. Records with the following MARSS characteristics are excluded from
the Graduation measurement:
• MARSS Status equals 1
• Grade not equal to 08 - 12
Additionally, students with the following State Aid Categories are excluded from the
Graduation measurement:
• 14 – Attending in another State
• 15 – Attending in MN but tuition paid by another State
• 16 – Shared Time
• 17 – Shared Time
• 18 – Shared Time/Tuition
• 25 – Adult Student
• 28 – Resident Attending Non-Public School
Determining the unduplicated count (using the last reported student record in the
MARSS system)
An unduplicated count of student records over multiple years is required. To do so, only
the last record reported for any particular MARSS Number is selected. All other records are
ignored in the Graduation measurement.
To find the last reported record for an individual MARSS Number, the selected records
are evaluated in order by Fiscal Year and then the Status End Date. If two records for the same
MARSS Number end on the same date, the lower Status End code is used. If both have the same
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 74
status end code, the record with the higher documentID is used. Only that final record is
included in the computation with its corresponding demographics and status end code (graduate
or dropout code). The computation uses the following MARSS elements when selecting records:
From the student table:
• Submission Code
• Fiscal Year
• Status End Date
• MARSS Number
• Grade
• Status End Code
Once the last record is determined, the status end codes are then evaluated. Only those
records with a graduation or dropout code are retained. All other records are ignored.
Graduate Status End Codes
• 08, 09
Dropout Status End Codes
• 06, 14, 15, 16, 17, 18, 19, 31, 32, 33, 34, 35, 37
General Formula
To compute the entity’s Graduation rate for a particular year, the count of graduates in
that year is divided by the count of graduates plus selected dropouts from that year and previous
three years. Multiply this by 100 to provide the percentage where applicable. The most recent
fiscal year is referred to as Year 4 while the previous fiscal years are referred to as years 3, 2, and
1.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 75
Numerator
The count of students graduating in Year 4.
The computation uses the Fiscal Year in each selected MARSS record to determine if it
should be included in the count. If computing the Graduation Rate for 2005, the count of
graduates reported in 2005 is used as the numerator. Graduates from any other year are not
included.
Denominator
Count of grade 9 students dropping out in year 1
+ Count of grade 10 students dropping out in year 2
+ Count of grade 11 students dropping out in year 3
+ Count of grade 12 students dropping out in year 4
+ Count of students graduating in year 4
Only those dropouts reported in certain grades in certain years are included in the
denominator with the graduates. Dropouts or graduates from any other year are not included.
Formula Example
This example is illustrates how the Graduation rate for 2004 is computed.
• MARSS EOY data 2001 through 2004 is evaluated with MARSS Fall data from
2005.
• Only grade 8 – 12 records are evaluated eliminating certain conditions and state
aid categories described above.
• The last record for any particular MARSS number is found by using Fiscal Year
and Status End Date – all other records for that MARSS number are removed.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 76
• Only records where the Status End Code is either a graduate code or a dropout
code are used – all other records are removed.
• The remaining records are evaluated counting the graduates and dropouts within
each entity.
Count of Students Graduating in 2004
Count of grade 9 students dropping out in 2001
+ Count of grade 10 students dropping out in 2002
+ Count of grade 11 students dropping out in 2003
+ Count of grade 12 students dropping out in 2004
+ Count of students graduating in 2004
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 77
APPENDIX F
External Auditing: Utah
Utah External Auditing
2.1. To assure users of aggregate membership, fall enrollment, and dropout data reported
by districts to the USOE via the Data Clearinghouse that such data are reasonably accurate and
supported by adequate local documentation. To identify sources of errors in recording and
reporting nonfiscal data for the purpose of making continuous improvements to the quality of
student accounting systems.
STANDARDS FOR RECORDING AND REPORTING STUDENT DATA
3.1. Standards for organizing and maintaining a student accounting system and for
reporting pertinent to these agreed upon procedures are found principally in two documents:
3.2. State Board of Education rule R277-419 on “Pupil Accounting” contains the legal
standards and is found at http://www.rules.utah.gov/publicat/code/r277/r277-419.htm.
3.3. The USOE “Data Clearinghouse Update Transactions” file layout contains the
technical standards and is found at
http://www.schools.utah.gov/computerservices/Clearinghouse/Clearinghouse.htm.
3.4. The following parts of the Data Clearinghouse document are the most relevant to the
purposes of these agreed upon procedures:
• S1 Exit Code field — this is the reason why the student left school before the end of
the school year
• S1 School Membership field — this is “regular” membership
• S2 SCRAM Membership field — this is special education membership
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 78
• S2 SCRAM Time/Setting field — note the distinction between “self contained” and
“resource” types of special education students
In addition to reading the above mentioned documentation, the independent accountant
should also become familiar with the district’s data management policies and practices,
especially as these impact the district’s production and submission of the Year End and Fall
Clearinghouse files to the USOE.
AGGREGATE MEMBERSHIP:
4.1. Select schools in the district such that each school is included in the sample at least
once every five years; if feasible, a shorter cycle, such as once every three years, would be
preferable.
4.2. Visit each school in the sample.
4.3. Select students in the schools such that the total number of students in the sample is
equal to or greater than the following size according to the enrollment of the district on the
previous October 1:
Enrollment Sample Size
40,000 or greater 70
20,000 to 39,999 50
10,000 to 19,999 40
1,000 to 9,999 30
Less than 1,000 20
Students sampled should include an appropriate representation from each compliance rule
[see 4.4(a) through (g)] with a focus on potential or identified risk of noncompliance, i.e.,
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 79
students who enter or exit during the school year and students with significant absences.
4.4. For each student in the sample, study the student’s records and supporting
documentation (particularly for entry and exit dates), identify the student’s aggregate
membership as recorded in the records, and determine whether the following rules from R277-
419 were properly applied in calculating the student’s aggregate membership:
(a) Ten Day Rule [R277-419-1(O)] If the student had 10 consecutive school days
of unexcused absences, the student’s exit date is not later than the school day after the 10th
day of
such absences and, consequently, the student did not generate membership from that day on. An
“unexcused absence” means an absence charged to a student when the student was not physically
present at school at any of the times attendance checks were made during the day in accordance
with R277-419-3(B)(4), and the student’s absence could not be accounted for by evidence of a
legitimate excuse in accordance with the local board of education policy on truancy as defined in
Utah Code 53A-11-101.
(b) Maximum 990 Hours Rule [R277-419-3(A)] This comprises three related
equations: (i) the sum of the student’s regular (K-12) membership and special education self
contained membership does not exceed 180 days; (ii) the sum of the student’s special education
self contained and special education resource membership does not exceed 180 days; and (iii)
the sum of the student’s regular membership and special education resource membership does
not exceed 360 days.
(c) Individualized Education Plan Rule [R277-419-3(B); R277-750]
If the student was enrolled in a special education program, there is an appropriately completed
IEP for the student justifying the service.
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 80
(d) Homebound/Hospitalized & Suspension Rule [R277-419-4(C)] If the student was in
membership while homebound, hospitalized or suspended, (i) the student received a minimum of
two hours of instructional contact by a licensed educator each week and
(ii) the circumstances requiring this instructional arrangement are clearly documented.
(e) Part Time Proration Rule [R277-419-4(E)(1)] If the student was enrolled for
only part of the school day and/or only part of the school year, the student’s membership was
prorated according to the number of hours or periods the student was actually enrolled in relation
to the number of hours or periods the student could have been enrolled. As one example, if the
student was in membership 4 periods each day in a 7-period school day for all 180 school days,
the student’s aggregate membership is 103 days; as another example, if the student was in
membership for 7 periods each day in a 7-period day for 103 school days, the student’s
membership is 103 days.
(f) Released Time Rule [R277-419-4(E)(2)] If the student was released for
religious instruction or individual learning activity, (i) there is a Student Education/Occupation
Plan (SEOP) signed by the student, the student’s parent or guardian, and a representative of the
school indicating the use of released time for this purpose is consistent with the plan and (ii)
released time did not exceed the equivalent of one period per school day.
(g) Youth In Custody Rule [R277-419-4(G)]
If the student was enrolled in YIC classes for two to four hours a day (YIC Program Code = ISI
1), regular membership cannot exceed half of total possible regular membership for the student.
If the student was enrolled in YIC classes for more than four hours a day (YIC Program Code =
ISI-2), regular membership must be zero.
4.5. For any student whose reported aggregate membership is based on a violation of one
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 81
or more of these rules, recalculate aggregate membership to determine the correct figure.
4.6. Use the appropriate illustrative report at the end of this document as a model for
writing a formal summary of your findings and report complete details of your findings for each
student in the sample in the format specified under the “Aggregate Membership” tab of the APP
C-5:
TRANSFER STUDENT DOCUMENTATION:
6.1. Select secondary schools in the district such that each school is included in the
sample at least once every four years; if feasible, a shorter cycle, such as once every three years,
would be preferable. These may be the same secondary schools which were selected for the
purpose of applying agreed-upon procedures for Fall Enrollment in 5.1.
6.2. Obtain a copy of the Transfer Students List (from the prior Year End upload of the
Clearinghouse), which contains students, organized by school, who were:
(a) enrolled in grades 7 through 12; but
(b) not enrolled on the last day of the school year; and
(c) not classified by the district as either high school completers or dropouts.
6.3. Select a sample of students from this list equal to or greater than 5% of the total
number students on the list; however, the sample size should not be fewer than 10 and need not
exceed 30.
6.4. For each student in the sample, determine whether adequate documentation exists to
support the district's claim that the student was not a dropout. The following constitute adequate
documentation for the possible types of students on the list:
(a) of “transfer” to another school (TD, TO, TP, TS) — an official request for the
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 82
student’s records by the receiving school as evidenced in a formal written communication from
the school; or an entry in a log systematically maintained for the purpose showing either (i) the
name of the student, the name of the person and the name and address of the school making the
request, and the date the request was made; or (ii) especially in cases where the parent requests
the student’s record at the time of exiting, an effort by the school to confirm the reenrollment of
the student in another school.
(b) of “transfer” out of the United States (TC, perhaps some instances of TO) — a
statement by a member of the community who has personal knowledge that the student has
moved to another country.
(c) of “transfer” to home schooling (TH) — evidence that the student was issued
a certificate exempting them from public school attendance for the purpose of home schooling in
accordance with Utah Code 53-11-102.
(d) of “withdrawal” (WD) — evidence based on a source external to the school
explaining the student’s situation and justifying withdrawal without continuing provision of
educational services in accordance with R277-419-4(C) as a reasonable response.
(e) of “death” (DE) — a copy of the death certificate or an obituary as published
in a commercial newspaper.
6.5. Use the appropriate illustrative report at the end of this document as a model for
writing a formal summary of your findings and report complete details of your findings for each
student in the sample in the format specified under the “Transfer Student” tab of the APP C-5:
Sample Schedules spreadsheet available from Emily Eyre.
Do not estimate dropout counts for any school or for the district as a whole. Any
adjustments to dropout counts in light of the compliance findings for the purpose of
Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 83
accountability reporting will be made at the discretion of the USOE.
top related