data-driven decision-making for quality control:
DESCRIPTION
TRANSCRIPT
![Page 1: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/1.jpg)
Data-Driven Decision-Making for Quality Control:
The Power of a Relational Database
Vicki L. Cohen, Ed.D.Marlene Rosenbaum, Ed.D.Joshua Cohen
Fairleigh Dickinson UniversityTeaneck, NJ 07666
Presented at the Annual AACTE ConferenceNew Orleans, February 2008
![Page 2: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/2.jpg)
This session will: Describe how using a relational database
becomes the driver of a quality control system;
Describe the development and utilization of a relational database;
Show how data is leveraged to support student learning, program revision, and outcomes based assessment.
![Page 3: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/3.jpg)
The School of Education (SOE) at Fairleigh Dickinson University Comprised of aprox. 1,000 students Multiple programs that apply for state
certification: 5-Year Accelerated QUEST program MAT LD Educational Leadership Reading Specialist
![Page 4: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/4.jpg)
The SOE at a Glance On two campuses (Teaneck and Madison) Located at 3 Community Colleges throughout
the state of New Jersey 15 Full-time faculty members Aprox 35 part-time faculty Place approximately 120 candidates into
student teaching/year Place a total of approximately 700 candidates
into clinical field experiences/year.
![Page 5: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/5.jpg)
FDU at a Glance
FDU has aprox 12,000 students Largest private university in State of NJ SOE is part of University College on Teaneck
campus QUEST program:
45 candidates at Teaneck 200 Madison 75 CC
![Page 6: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/6.jpg)
SOE Is a Complex Program!
Went for TEAC accreditation
![Page 7: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/7.jpg)
Preparation for TEAC School of Education (SOE) needed to collect
accurate information on its claims Started gathering data on programs and student
performance Recognized need to access, organize and
analyze data in meaningful ways Developed a relational database This would become the “driver” of our Quality
Control System
![Page 8: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/8.jpg)
The Quality Control System (QCS)
Every institution and program has a set of procedures and policies to ensure quality in hiring, admissions, curriculum, program design, and student learning.
Together, these procedures and structures function as a Quality Control System (QCS). (TEAC)
![Page 9: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/9.jpg)
Need for Valid and Reliable Data
QCS must yield valid and reliable evidence about the program’s practices and results, which influences its policies and decision making.
![Page 10: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/10.jpg)
What is Valid Evidence? Are we measuring what we intended to
measure? Are we sure that our evidence is pointing
us in the right direction? How confident do we feel about the data
we collected?
![Page 11: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/11.jpg)
“Am I measuring what I think I am measuring?”
![Page 12: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/12.jpg)
What is Reliable Evidence?
Yields results that are accurate and stable
Collected in a consistent way Confident we we are making the right
decision.
![Page 13: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/13.jpg)
SOE TEAC Process
1. Developed a QCS that ensured we are collecting valid and reliable data on our claims and cross-cutting themes
Instruments (rubrics, observation forms, surveys) were validated through external panel of “experts”
Inter-rater reliability is being established
2. Developed infrastructure and system to collect the data
![Page 14: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/14.jpg)
SOE TEAC Process
3. Analyzed data: aggregated and disaggregated
4. Determined strengths and weaknesses with total faculty involvement
5. Analyzed what revisions needed • Programs• Curriculum• Processes and policies
6. Currently making revisions based upon evidence.
![Page 15: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/15.jpg)
SOE Assessment Philosophy
We use multiple sources of data that are designed to assess the performance of teaching candidates as they progress through our program.
Collect data in three areas.
![Page 16: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/16.jpg)
1) Throughout the Program
Continuous assessment of teaching candidates throughout the program from entrance, midpoint, and exit Grades GPA Praxis scores Rubrics Reflections
![Page 17: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/17.jpg)
2) In Clinical Field Experiences
Assessment of pedagogical knowledge and skills that occurs during clinical field experiences; Placement Observation forms
![Page 18: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/18.jpg)
3) Completion of Program
Perceptions of teaching candidates and alumni after they have completed their program, which is used for program improvement Alumni Surveys Exit Surveys Focus Groups
![Page 19: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/19.jpg)
Traditionally….Data Deficient
Schools of Education have not been collecting data systematically
Infrastructure not set up Not able to access multiple
sources of information
![Page 20: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/20.jpg)
Traditionally…..Data Dummies
What data do we want to collect? How can we manage it? What do we do with data? How do we organize it? Access it? Make sense?
![Page 21: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/21.jpg)
Currently….Data-Driven
Systematically collecting data Infusing into our faculty culture Meeting regularly to assess evidence Making decisions
based upon evidence Beneficial process
due to TEAC
![Page 22: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/22.jpg)
Data-Driven Decision-Making
Plan
CollectData
Impl
emen
tA
nalyze
Data
Reflect
& Revise
![Page 23: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/23.jpg)
Example of Data Collection:Praxis Results 2005/2006
Program FDU SOE Pass Rate
NJ Pass Rate
Elementary Ed Content Knowledge 99% 84%
English Lang Lit Content Knowledge 100% 71%
Social Studies Content Knowledge 100% 64%
Secondary: Math Content 100% 92%
Middle School Math 100% 65%
General Science Content Knowledge 100% 78%
Spanish Content Knowledge 100% 87%
![Page 24: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/24.jpg)
Mean Scores of Teaching Candidates on Selected CCI Indicators Related to Pedagogy as Rated by Supervisors (Fall 2005, Spring 2006)
Midpoint Final
Indicator N Mean Standard Deviation
N Mean Standard Deviation
2.2 Creates and implements lessons that are developmentally appropriate.
57 4.47 .630 51 4.75 .483
3.2 Effectively incorporates multicultural information and strategies when appropriate into the lesson; presents issues from a multicultural perspective.
57 2.82 2.010 51 3.59 1.846
4.1 All essential components of a well designed plan for coherent instruction.
57 4.58 .533 51 4.71 .672
4.2 Appropriate instructional objectives that are clearly stated, measurable and aligned with the NJCCCS.
57 4.42 .706 51 4.67 .476
4.3 Effective use of a wide range of resources including technology to enhance instruction and student learning.
56 3.86 1.420 51 4.39 .918
4.4 Demonstrates understanding of curriculum design and implementation, including various curricular approaches (i.e. interdisciplinary, integrated, thematic).
57 3.79 1.278 51 4.12 1.336
![Page 25: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/25.jpg)
Wanted: Database Administrator
New Job Description: full-time professional staff
Resources Support from administrators Ensure candidate had appropriate
knowledge and skills to design database
![Page 26: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/26.jpg)
What Is a Database System? A collection of data organized in tables,
which can be accessed and manipulated, without having to restructure the tables
Elements of a Database System• A storage system • Data structures• Manipulation tools
![Page 27: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/27.jpg)
Database Advantages Analyze sophisticated correlations easier
because relationships are established between data sets
Make decisions based on information derived from data
Streamline business operations Organize data and eliminates:
• Inconsistent data• Missing data• Redundant data
![Page 28: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/28.jpg)
Problem #1: Field Placement Office
Staff was overwhelmed with managing Clinical Placements
In 2007 800+ letters were mailed to 424 schools asking for placements
Previously, these letters were individually prepared in MS Word documents
![Page 29: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/29.jpg)
Problem #1: Field Placement Office (cont’d)
Clinical Placements must be coordinated with• School• Student• Supervisor
Difficult to aggregate data• What districts have most confirmed/declined rates?• What trends are we seeing?• What kind of schools are we sending our candidates to?
![Page 30: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/30.jpg)
Solution #1: Streamlined Field Placement Office Developed a data collection system for
Clinical Placements Data is:
• Entered on 2 campuses• Used to create personalized communications to
schools, students, and supervisors• Used to manage Clinical Placements
• Confirmed/Pending/Declined• Supervisor Assignments
![Page 31: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/31.jpg)
Solution #1: Streamlined Field Placement Office (cont’d)
Gives us the ability to aggregate data, look at trends, and make decisions Confirmed / Declined distribution by district Analyze demographics of cooperating school
districts
![Page 32: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/32.jpg)
Solution #1: Infrastructure
![Page 33: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/33.jpg)
We start with the person record from the university system
![Page 34: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/34.jpg)
Data on the Clinical Placement is entered
![Page 35: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/35.jpg)
Solution #1: Generating letters
![Page 36: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/36.jpg)
Generate standardized reports for “master lists”
![Page 37: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/37.jpg)
Solution #1: Payroll
![Page 38: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/38.jpg)
Problem #2: What Type of Districts Do
We Place Our Candidates Into?
The District Factor Group (DFG) is a socioeconomic indictor used for comparative test reporting of school districts for New Jersey’s statewide programs.
![Page 39: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/39.jpg)
Problem #2: What Type of Districts Do We Place Our Candidates Into? (cont’d)
DFG Factors: % of adult residents failed to complete high school % of adult residents who attended college Occupational status (laborers, service workers, farm
workers, professionals, etc.) Population density Income Unemployment rate Poverty rate
![Page 40: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/40.jpg)
Problem #2: What Type of Districts Do We Place Our Candidates Into? (cont’d)
Eight DFGs have been created based on the 1990 United States Census data
Range from A (lowest socioeconomic district) to J (highest)
A, B, CD, DE, FG, GH, I, J
![Page 41: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/41.jpg)
DFG: State DistributionDistrict Factor Group Distribution
For All Districts In NJ
0
20
40
60
80
100
120
A B CD DE FG GH I J
![Page 42: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/42.jpg)
DFG: Apprenticeship Teaching Distribution
Distribution For Apprenticeship Teaching In Spring 2007
0
5
10
15
20
25
30
35
District Factor Groups
Fie
ld P
lac
emen
ts
A B CD DE FG GH I J
![Page 43: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/43.jpg)
DFG: Alumni DistributionReported Distribution of Working Alumni
0
2
4
6
8
10
12
14
16
18
District Factor Groups
Alu
mn
i
A B CD DE FG GH I J
![Page 44: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/44.jpg)
Solution #2: Share Data, Discuss, Revise Evident discrepancy between where alumni get
jobs and where candidates are placed We share evidence with faculty and key
stakeholders They discuss and make appropriate decisions
![Page 45: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/45.jpg)
DFG: In the future
Will have full record of where candidates performed clinical experience
Will have record of where they are working
Can correlate accordingly
![Page 46: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/46.jpg)
Data Needed
A state database of teacher employment Difficult to track alumni as they move from
school to school Unique teacher & school identifier
State database needs to integrate with University and commercial marketing data systems
![Page 47: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/47.jpg)
Problem #3: Data Systems Not Integrated
SOE’s recordkeeping is not integrated with the University system Student information is entered into SOE
system manually Limits power of reporting Duplicate person records may exist if Student
ID is not entered correctly
![Page 48: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/48.jpg)
Problem #3: Data Systems Not Integrated (cont’d)
A “live” data connection to University system is not possible Technology is not in place Support for integration is needed
![Page 49: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/49.jpg)
More Efficient and Effective Use of Resources
Relational database assists with: Streamlined SOE business operations
Generate mail merge letters Provide reports Automate payroll
Leverages existing data to create information for program improvement
![Page 50: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/50.jpg)
Started Slowly
Started with trying to code and track our students properly
Administrative assistant created rudimentary Access Database
When she left, we hired a consultant to manage database
He totally redesigned and reorganized it
![Page 51: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/51.jpg)
Skill Set for Database Administrator Problem-solving Relational database design skills SQL proficiency Knowledge of structured programming
language Excellent communication skills Work with faculty and technical staff “People-skills”
![Page 52: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/52.jpg)
Conducted Extensive Search Advertised Set up search committee Interviewed many different applicants Required each applicant to take a test Presented problems to candidates to
assess problem-solving abilities Found many could enter data, but not
design or problem solve
![Page 53: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/53.jpg)
Working with the University
Collaborating with the Arts and Sciences Establishing knowledge-base in content
areas and general education Addressing NJCCCS and Professional
Standards in discipline Aligning content courses and standards in
matrices
![Page 54: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/54.jpg)
Working with the University(cont’d)
Meeting with individual departments Establishing long-term relationships Shifting to new paradigm--Learning Outcomes
Assessment Working with A&S to collect data Database Administrator playing key role in
collection of data across college
![Page 55: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/55.jpg)
Leveraging TEAC Across the University Establishing the need for LOA: Middle States Educating the A&S faculty: LOA process Addressing resistance of A&S faculty Establishing a relational database system for
university: program, college Creating the infrastructure to collect data Collecting multiple sources of data for A&S Getting various groups to communicate and
plan.
![Page 56: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/56.jpg)
Conclusion: The Power of a Relational Database
Data-driven Decision-making requires
an integrated system of collecting data from many different sources.
![Page 57: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/57.jpg)
Systemic The whole institution needs to be
vested in the collection of data Data needs to be collected on faculty,
students, courses, grades, scores, rubrics, observations
University and Colleges need to ensure data collection systems are in place early.
![Page 58: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/58.jpg)
Integrated
All data sets need to be connected so relationships can be established
Queries made Reports generated Correlations and relationships analyzed
![Page 59: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/59.jpg)
Benefits for Total University Middle States Accreditation Nursing Engineering College of Business Program improvement Student learning
![Page 60: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/60.jpg)
School of Education Leads the Way
Ultimate goal is to improve teacher quality and impact achievement for all students.
Data provides the means to do this. Relational database is the engine
that makes this possible.
![Page 61: Data-Driven Decision-Making for Quality Control:](https://reader033.vdocuments.us/reader033/viewer/2022061200/54743e56b4af9fa90a8b55ce/html5/thumbnails/61.jpg)
For more information please contact:
Vicki L. Cohen [email protected], School of Education
Marlene Rosenbaum [email protected] Dean, University College
Joshua Cohen [email protected] Administrator