bpc-ncwit evaluation workshop boulder, co may 16, 2007 evaluation of the bpc alliance projects daryl...

BPC-NCWIT Evaluation WorkshopBoulder, COMay 16, 2007

Evaluation of the BPC Alliance Projects

Daryl E. Chubin

AAAS Capacity Center

www.aaascapacity.org

BPC-NCWIT Evaluation - Boulder 05-16-07

AAAS Portfolio Assessment of BPC Alliances:Purpose

• The portfolio assessment will look across the Alliance projects and ask—both in knowledge gained and impact on participants—what does investment in the set yield?

• The portfolio will be constructed from the data collected and reported by the projects.

• AAAS will offer technical assistance to the project evaluators, while framing the respective efforts relative to the overall BPC mission.

• AAAS’s purpose is ultimately to help “tell the story” of BPC.


Guiding Questions

• How is BPC affecting the capacity of the community to broaden participation from education to workplace?

• What practices seem promising for whom and in what settings?

• How are relationships among partners maturing, being leveraged, and expanding the sphere of influence within the national computing community?


Constraints

• Seeking short-term outcomes from long-term interventions

• Measuring processes and organizational arrangements as well as readily-quantified outputs (e.g., enrollees, graduates, achievements)

• Admitting what is seemingly not working as well as what is—and describing adjustments

• No AAAS access—yet—to Alliance annual reports

• Special scrutiny of programs by the Administration due to the Academic Competitiveness Council report


Definition of Success for the Academic Competitiveness Council (2006)To fulfill the requirements of the Deficit Reduction Act, the council

will:• Identify all federal education programs with a STEM focus;

• Identify the target populations served by these programs;

• Identify the stated goals of these programs;

• Identify the extent to which the programs have undergone independent external evaluations based on sound, scientific principles;

• Identify the extent to which the programs have quantitative evidence of achieving stated goals;

• Identify or develop standards for measuring and evaluating these programs, including common measures as appropriate; and

• Recommend administrative or legislative action to efficiently integrate and coordinate federal spending on STEM education programs.

Source: Report of the Academic Competitiveness Council, May 2007


NSF/BPC Alliance Projects: Strategies


NSF/BPC Alliance Projects: Populations Targeted


Suggested Variables for Baseline Data-Collection*

Suggested Variable Illustrative Disaggregations

Study Population gender, ethnicity, disability, education level/cohort, institutional affiliation

Institutional Partners number, sector (e.g., academic, corporate, nonprofit), forms of participation (i.e., division of labor) Origin of Intervention number of pre-existing programs (e.g., enrichment, mentoring, internship), ways of distinguishing BPC

participants from others (recruits, applicants, enrollees, majors, etc.)

Change in Study Population attitudes, enrollment, performance (e.g., class/project grade, GPA, retention, skills/knowledge, aspirations, degree completion)

Change in Strategy added/dropped activity, distribution of time/effort in activity, acquisition of new partner, addition of expertise on project staff

*see BPC Alliance Methods of Evaluation (handout)


Evaluation of the BPC Alliance Projects

Agenda

10 AM Cohort 1 project presentations of baseline data (5-10 minutes

each)

11 Cohort 2 project overviews of baseline data (5 minutes each)

11:30 Discussion of common needs, challenges, & strategies for

data- collection, analysis, & interpretation

bpc-ncwit evaluation workshop boulder, co may 16, 2007 evaluation of the bpc alliance projects daryl...

Documents