placement testing, basic skills, and student...
TRANSCRIPT
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Dr. Darla M. Cooper
Director of Research
and Evaluation
The RP Group
What You Need to Know as a Researcher
Placement Testing, Basic Skills, and Student Success
2011 RP Summer Institute
August 8: 1:15 p.m. – 3:15 p.m.
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Learning Outcomes
• Identify the key resources available, why
you would want to use them and where can
you find them
• Recognize the role of the researcher in
placement testing, basis skills, and student
success
• Explain key information related to placement
testing, basis skills, and student success
2
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing
3
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Matriculation Regulations
• Each college district to establish a program
of institutional research for ongoing
evaluation of its matriculation process to
ensure compliance with the regulations.
• Part of the evaluation is to ensure that tests
and procedures minimize cultural or
linguistic bias and are used in an
appropriate manner.
4
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Matriculation Regulations (cont’d)
• Test publishers and colleges using “locally
managed instruments” must renew approval
every 6 years. They should collect renewal
evidence sometime between 4th and 6th years,
and submit renewal evidence during 5th year.
• Local college using second party instruments still
must update their evidence on content-related
validity, cut score validity, and
disproportionate impact during the renewal
period. This updated evidence must be kept on file
for review during the matriculation on-site review.
5
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing
• Primary function is to “assist the student in
making decisions about appropriate course
level enrollment, major area of study, and
vocational program choice.”
• List of approved assessment instruments is
published at least annually by the CCCCO
2008-09 Approved Assessment Instruments
6
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Validation Study
• “Through the validation process, the
college/district assures that it is employing
scores and tests that are appropriate and
fair for all students based on the standards
developed for community college placement
instruments.”
• Most common request of researchers
7
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Validation Study Components
• Content Validity (Design 14)
• Measures the extent to which course prerequisite
knowledge and skills are measured by the test
• Faculty complete a matrix that matches test to
prerequisite skills
• Disproportionate Impact (Design 12)
• Helps ensure that placement test minimize or
eliminate cultural or linguistic bias and are being
used in a valid manner based on ethnicity,
gender, age or disability
8
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Validation Study Components
• Cut Score Validity
• Adequacy of cut scores is demonstrated by
either a judgmental or empirical approach
• A judgmental approach typically is used
solely for initial setting of cut scores
• Empirical evidence can be of two types:
criterion-related validity evidence or
consequential-related validity evidence
9
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Validation Study Components
• Criterion Validity (Design 10)
• Criterion done before test implementation
• Measures the extent to which scores on the
placement test are related to results of an
appropriate criterion measure of student
ability
• Criterion can be exam scores, instructor
ratings, grades, etc.
10
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Validation Study Components
• Consequential Validity (Design 15)
• Consequential done after test implementation
• Evaluates the appropriateness of students’
placement into the course
• Compares an instructor’s evaluation of the
readiness of individual students to undertake
the material of his/her course, and each
student’s evaluation of the appropriateness of
his/her placement into that course.
11
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Resources
• Key resources on CCCCO website:
• Matriculation Research Monographs
• Guide to Evaluate Tests
• Assessment Q&A
• Academic Senate website gives insight into
faculty perspective, for example:
• Student Success: The Case for Establishing
Prerequisites through Content Review
• RP website
• Placement Testing and Prerequisite Validation
12
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Resources
• Research Monographs • Single most important resource
• Explains and provides examples for validation
studies
• Assessment Validation Project Local Research
Options
• Matriculation Local Research Options Project
• Matriculation Evaluation: Monographs on Designs
from the Local Research Option Programs
• Matriculation Evaluation: Phase III Local Research
Options
13
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Role of the Researcher
• Be familiar with the relevant requirements
and regulations
• Know the validation process
• Perform statistics and analyses
• Facilitate interpretation of results
• Remember that faculty are the discipline
experts and, as a researcher, you are
serving them.
14
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Role of the Researcher
• Be prepared to answer other questions
related to placement testing such as:
• Do students who place into a course perform
better than students who advance to the
course from the prerequisite?
• Do students who have completed college-
level English perform better in writing
intensive courses such as history,
anthropology, communication, and sociology?
15
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Case Study How validation studies can lead to positive changes for students
16
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Placement Testing Case Study Discussion Questions
1. Why do you think this situation resulted in a
successful outcome for students?
2. If you found similar results in a validation
study, what would you do? Who would you
go to? What would be your approach?
17
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills
18
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Resources
• Basic Skills ARCC
• Poppy copy
• BSOC – recommended measures
• BRIC Inquiry Guide – Assessing Basic
Skills Outcomes
• Academic Senate Basic Skills website
• California Community College Success
Network (3CSN)
19
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Role of the Researcher
• Be familiar with basic skills at your college
• Advocate the importance of evaluation
• Cooperatively develop evaluation plan
• Bring expertise on research design and
methodology
• Perform statistical analyses
• Assist with survey design and analysis
• Facilitate interpretation of results
• Ask questions
20
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Role of the Researcher
Questions to ask: • What would you want students who
participate to gain?
• How would you know they achieved these desired outcomes?
• How do you think you could measure these desired outcomes?
• What else do you want to know about students’ participation and how can you obtain this information?
21
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Evaluation • Usually centers around the
implementation of interventions and
answering the question:
• How do you know interventions (programs)
are working?
• Includes both formative and summative
evaluation
• Uses direct measures whenever possible
• Researcher usually asked to play the role
of evaluator
22
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Evaluation • What are the common interventions?
• Tutoring
• Learning communities
• Supplemental instruction
• First year experience
• Professional development
• What are the common methods?
• Pre/Post Test
• Group Comparisons
• Trend Analysis
• Surveys
• Qualitative
23
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Activity Learning Communities Evaluation
24
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Learning Communities Evaluation
The Basic Skills Committee has come to you
for help with evaluating the impact of learning
communities on basic skills students. They
would like you to suggest possible evaluation
measures that could be pulled together into
an overall evaluation of learning
communities.
25
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Learning Communities Evaluation
Definition
• Students take 2-3 linked courses where the
faculty of each course work together to
coordinate curriculum, assignments,
assessments and grading
Example
• Three classes are linked in the LC: History 101,
English 90 (one level below transfer), and
Counseling 50 (College Readiness)
26
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Learning Communities Evaluation
In small groups, answer the
following questions:
1. How would you approach this
request?
2. What measures might you suggest
for this evaluation?
27
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Evaluation Illustration – Learning Communities
Group Comparisons (Summative)
Compare success rates between LC students and
students in the same course outside the LC to
demonstrate that the LC helped students achieve
greater success
• Example: Compare success rates of students in LC
English 90 to students in other non-LC sections of
English 90
28
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Evaluation Illustration – Learning Communities
Group Comparisons (Summative)
Compare persistence rates to the next course in the
sequence between LC students and students in the
same initial course outside the LC to demonstrate
that the LC helped students persist and succeed at a
greater rate
• Example: Compare percentage of students in LC
English 90 who persisted to and succeeded in
English 101 (transfer level) the following semester to
students in other non-LC sections of English 90
29
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Evaluation Illustration – Learning Communities
Pre-/Post-tests (Summative)
Assess students’ skills/knowledge, and/or abilities at
the beginning and end their participation in the LC to
demonstrate that participation resulted in
improvement
• Example: Assess students’ knowledge and skills
related to college readiness at the beginning and end
of the Counseling 50 course
30
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Basic Skills Evaluation Illustration – Learning Communities
Surveys and Interviews (Formative)
• Survey students at the end of the semester
to assess their experience in and
satisfaction with their participation in the LC
• Interview faculty teaching the linked courses
to assess what worked and did not work and
their perceptions of students’ experience
and the impact of LCs on student learning
31
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Student Success
32
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Student Success Quantitative Measures
The Usual Suspects aka “The Rates”
• Course success rates
• Persistence rates
• Degree/certificate rates
• Transfer rates
• ARCC SPAR
33
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Student Success Resources
• Data on Demand
• TVP tool
• Institutional Research Operational
Definitions
• Data 101: Guiding Principles for Faculty
(Academic Senate)
• Guiding Principles for SLO Assessment
(Academic Senate)
34
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Student Success Other Measures
Expand Your Mind (and Your
Definition of Student Success)
• More in-depth quantitative
• Qualitative methods
• Student learning outcomes
35
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Student Success More In-Depth Quantitative
• Go beyond the rates
• Ask deeper questions
• Find meaningful ways to look at the rates
• Example
• Three reports to energize your student
success discussions
36
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Student Success Qualitative Methods • Often asks the question of “how” and “why”
instead of “what”
• Focuses on the details; more holistic
• Looks at the quality of relationships, activities, experiences, situations, or materials
• Types of methods
• Participant observation
• Direct observation
• Interviews
• Focus groups
• Case studies
37
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Student Success Student Learning Outcomes
• Focus is on what students learn not what
they achieve
• Identifies what students are expected to
learn and specifies how to measure that
learning in order to demonstrate that the
learning has occurred
• Emphasis is more on analyzing the findings
and determining needed action, not the
findings themselves
38
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Student Success Role of the Researcher
• Help define and measure student
success
• Bring expertise on quantitative and
qualitative research design and
methodology
• Perform statistical analyses
• Assist with survey design and analysis
• Facilitate interpretation of results
• Ask questions
39
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Student Success Role of the Researcher
Special note on researchers and SLOs
• Depends on the faculty need and
researcher expertise
• Primarily consultative role, usually only
upon request
• Bring expertise on assessment
techniques, research design and
methodology
• Perform statistical analyses
40
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Student Success Role of the Researcher
Special note on classroom-based research
• Empowers faculty to answer their own
questions
• Helps build a culture of inquiry
• Creates faculty ownership of the research
• Researcher can help:
• Refine research questions
• Introduce/review design and methodology
• Design survey instruments
• Analyze and interpret data
41
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Student Success Small Group Activity
A faculty member wants help with a classroom-
based research project:
1. Choose one of the three scenarios
2. Answer the following questions:
• Which research design would you suggest and why?
• What assessment(s) would you suggest and why?
• What analysis of the data would you suggest?
3. Report out by scenario
42
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Research is a Partnership
43
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Relationship between Faculty and Researchers
• This relationship is key in all three areas:
• Placement testing, basic skills, and student success
• Researchers need to understand what is being
studied
• Faculty, staff and administrators need to
understand demands on researcher
• Develop research/evaluation plan together
• Researcher is seen as a member of the team
• Ongoing relationship is key to ongoing success of
the research
44
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
How Researchers Can Help Faculty
• Provide options for assessment methods
• Share knowledge of data already available
• Facilitate accurate data interpretation
• Ask questions
• Listen
• Do NOT dictate data to be used
• Do NOT advocate changing program/project
to fit the data
45
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
What Researchers Need From Faculty • To be invited and included in the conversation
early
• Share details of the project/program (e.g., goals, data, interventions, intended outcomes)
• Consider options provided by researcher and provide constructive feedback
• To be kept involved and informed about ongoing progress and changes in the program/project
• Work with researcher to make the data meaningful and useful
• Patience and understanding
46
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
How to Help Your Colleagues Prepare to Meet You Ask them to prepare answers to the following questions:
1. Goal(s) – What effect do you intend the program/project to have?
2. Outcomes – What tangible results do you expect to see from students?
3. Intervention – What treatment(s) students will receive? Where and when are the points of contact with students?
4. Data – What data will demonstrate intended outcomes? What data are the program/project already collecting or planning to collect?
47
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Striking a Balance Between Reality and Rigor
• First identify data already being collected
• Data collection should not place an undue
burden on the program/project
• Use direct measures whenever possible and
reasonable
• Need to ensure that data being collected are
actually measuring what you intended to
assess
• Requires conversation between practitioners
and researchers to achieve a suitable balance
48
2011 RP SUMMER INSTITUTE: Placement Testing, Basic Skills, and Student Success
Contact Information
Dr. Darla M. Cooper
Director of Research and Evaluation
The RP Group
49