educational realities
DESCRIPTION
Alliances for Graduate Education and the Professoriate Comparison Groups and Other issues September 18, 2008 by Catherine M. Millett, Ph.D. Policy Evaluation & Research, ETS. Educational Realities. The Circumstances : Some may believe that graduate education is a luxury - PowerPoint PPT PresentationTRANSCRIPT
1
Alliances for Graduate Education and the Professoriate
Comparison Groups and Other issues
September 18, 2008
byCatherine M. Millett, Ph.D.
Policy Evaluation & Research, ETS
2
Educational Realities
The Circumstances:• Some may believe that graduate education is a luxury• Graduate degree holders could be viewed as an elite population –
Among people 25 and over 1.3% have a Doctorate degree and 7% a Master’s degree
• NSF receives 2.5% of the $166.5 billion federal budget for education in 2006
The Result:• All of us have to continually get out the word that graduate
education is important and AGEP is important.
3
About AGEP
• October 1998, NSF awarded 8 universities nearly $2.5 million each to significantly increase the number of African American, Hispanic and Native American students receiving SME degrees.
• As of June 2008, there are 28 AGEP & SBE alliances• AGEP seeks to determine the college and university
policies, practices, and support services that lead to increased numbers of minority SME undergraduates that then enter into graduate school and professorial careers.
4
Roosevelt’s Words of Wisdom
• 2 cycles of the program have been completed. What do you expect?
• Evolution of external perception– “This won’t work”– “AGEP isn’t working”– “While AGEP isn’t responsible for changes
it is making a difference”• Now is the time to take stock of what has
been accomplished and what needs to be accomplished.
5
Micro and Macro View of AGEP
AGEP
Alabama AGEPMichigan AGEP
AlliancePROMISE
Rice-Houston AGEP
Who can best tell the Macro successes?Who can best tell the Micro successes?
Macro View - AGEP Alliance Models:
Program Elements Alabama Michigan Rice Houston
Promise
Recruitment trips Yes Yes Yes Yes
On-campus multi-week orientation program
Yes
Mentoring Yes Yes Yes Yes
Annual student reviews Yes Yes
Financial support to attend professional conferences
Limited Yes Yes
Publications – sole authored, co-authored with faculty and students
Yes Yes Yes
Time to Degree - Complete degree plus/minus 1.5 years of national average within field
Yes Yes Yes
Earn a tenure track job at XXX? Yes Yes Yes Yes
• Common goal: increase degree recipients
5
FABRICATED DATA
7
Individual Alliance Reports
• Showcase the Alliance’s accomplishments – Veronica told us to “Tell Your Story”
• What is your AGEP student experience?• Think about the various audiences for your reports – NSF, public
policy makers, university administrators, other faculty, the public, prospective and current graduate students.– What are we paying for and why? (e.g. trips to conferences, faculty
recruiting trips?)
• Provide information to NSF and other evaluators that may not be in AGEP wide report:– Students’ presentations and awards– Press coverage– NSF employees participation in Alliance events– How you are leveraging AGEP funding
8
Figure 2 – Hierarchy of Study Designs for Evaluating the Effectiveness of a STEM Education Intervention, by Expected Distribution of Study Type
EXPERIMENTAL:
Such as Well-designed Randomized Controlled Trials
QUASI EXPERIMENTAL:
Such as Well-matched Comparison-Group Studies
OTHER DESIGNS:
Such as Pre-Post Studies, and Comparison-Group Studies without careful matching
Source: Coalition for Evidence-Based Policy, 2006U.S. Department of Education. (May 2007) Report of the Academic Competitiveness Council
9
Putting Your Date in Context - Graduate Education Sources
• Graduate Record Examinations – Guide to the Use of Scores 2207-2008.
• US Department of Education (NCES 2007-162). The Path Through Graduate School; A Longitudinal Examination 10 Years After Bachelors’ Degree
• US Department of Education (NCES2006-185). Student Financing of Graduate and First-Professional Education, 2003-2004: Profiles of Students in Selected Degree Programs and Part-Time Students
• Survey of Earned Doctorates: 2006 Doctorate Recipients from United States Universities: Summary Report
• http://nces.ed.gov/das/library/nedrc_tables.asp• Professional societies may have data.
10
Variations in Defining AGEP Population
Comparison Groups – The Goal Fill in the sentence: AGEP could be making a difference in….
Answer the question: Is your program making a difference?
Goal of using comparison groups – provide credible evidence that your program could be making a difference
The Ideal Members:• Graduate students who have not participated in AGEP• Graduate students who are similar to AGEP participants• Have characteristics that the research base demonstrates are important
relative to the intended outcomes of the program.
Random Assignment to AGEP Treatments• Do you have an adequate number of students to do random assignment?• Do faculty and administrators support random assignment (e.g. in
admissions?)• Could you randomly assign students to other experiences in the AGEP
program (e.g. half of students receive full conference funding versus the other half who get 50% of conference support).
12
Other issues to consider
Naturally Found Groups• Applicants accepted to your graduate program but not
supported by AGEP.• Graduate students from pre-AGEP years• Graduate students who participate in other educational
programs from other agencies (e.g. NIH, Energy).• Students in graduate schools who are not in the AGEP AlliancesPossibilities:• Students who attend summer orientation versus those who do
not.• Students who attend semester long series on writing research
papers compared to those who do not.• Students who take a particular academic class (.e.g. statistics)
versus those who do not.
13
Practical Issues
Securing Human Subjects Approval
Accessing data
Planning for staff time and ability
Incentives
Response rates
Cost – if you have an evaluation budget remember to report how it is being spent and if you need more talk to your program officer. All they can say is no.
Confidentiality in reporting data
14
Get the A for Effort!