November 27, 2006

Howard Grimes

NRC Doctoral Program

Assessment

Background

• Help universities improve the quality of these programs through benchmarking

• Provide potential students and the public with accessible, readily available information on doctoral programs nationwide

• Enhance the nation’s overall research capacity

The goals of the NRC Doctoral Program Assessment:

Background

1. Identifying WSU Doctoral programs which meet the NRC Criteria (38 programs submitted)

2. The collection of quantitative data through a series of questionnaires

•Institutional questionnaire•Program questionnaire•Faculty questionnaire•Student questionnaire

3. The collection of data on faculty publications, citations and research activity

4. Anchoring Study• This questionnaire will be administered to a

random sample of faculty in each NRC field• Faculty members who have NOT completed the

faculty questionnaire will NOT be permitted to be raters

The Assessment Consists of:

Faculty Questionnaire

Faculty will be contacted by Mathematica Policy Research, Inc. on behalf of NRC and asked to fill out

an online questionnaire.

• Information on doctoral graduates will be used in NRC’s effort to gather placement data

•Program quality responses will be used to identify and weigh key study variables in conjunction with the Anchoring Study

•Faculty information will be used for the citation analysis (locate and count faculty publications and citations)

• Failure to provide this information correctly typically results in undercounts up to 25%

• This undercounting could adversely affect your doctoral programs ranking

What Is Citation Analysis?

• ISI will be used for the science and social science citation analysis

• Not all scholarly journals are covered by ISI. A list can be found at: http://scientific.thomson.com/mjl/

• Arts and humanities analysis pending

• BUT, even in the sciences and social sciences, it’s impossible to obtain accurate results consistently without obtaining information from authors

Science Citation Index (SCI)

Measure of the influence of one’s work

D.Z. Smith, J.S. Jam, M.Z.-C. Fu, and Z.Z. Top, “Parameterization of Aerosol Circulation in Global Circulation Models,” Journal of Atmospheric Science, 27:388-412 (2005).

D. Smith, J.S. Jam, M.Z-C. Fu, and Z.Z. Top, “Dissemination of Aerosols in the Troposphere: Implications for Global Circulation Models,” Journal of Atmospheric Science, 24: 221-229.

Why?Problem #1: Name Variants

NRC Faculty Questionnaire

Under what names or variants of your name have you published books or articles in the past five years (e.g. Jane Doe, Jane H. Doe, J. H. Doe or other prior names)?

H Grimes HD Grimes

Problem #2: Name Ambiguity

Two different people!

Johnson MK, Elder GH, Stern MAttachments to family and community and the young adult transition of rural youth JOURNAL OF RESEARCH ON ADOLESCENCE 15 (1): 99-125 2005 Times Cited: 2  

(Monica Kirkpatrick Johnson, Washington State University)

Cunningham WA, Raye CL, Johnson MKImplicit and explicit evaluation: fMRI correlates of valence, emotional intensity, and control in the processing of attitudes JOURNAL OF COGNITIVE NEUROSCIENCE 16 (10): 1717-1729 DEC 2004 Times Cited: 10  

(Marcia K Johnson, Yale)

NRC’s Solution

• ZIP CODE will be used by NRC to resolve name ambiguities and variances

• But what if they don’t have all the zip codes?

• AU=Johnson MK AND ZIP=99164 10 articles found Times cited = 17    She actually has a

total of 15 articles with 74 citations!

Why Were 5 Articles With 57 Citations Missed?

• MK Johnson did not list Postdoc zip code

• MK Johnson did not know one Postdoc article was published under a different zip code

Why Were 5 Articles With 57 Citations Missed?

One zip code for MK Johnson was not listed on ISI data baseAddresses: Univ Texas, Dept Sociol, Austin, TX 78712 USA Univ N Carolina, Dept Sociol, Chapel Hill, NC USA

Publisher: AMER SOCIOLOGICAL ASSOC, 1307 NEW YORK AVE NW #700, WASHINGTON, DC 20005-4712 USA

Subject Category: EDUCATION & EDUCATIONAL RESEARCH; SOCIOLOGY

IDS Number: 489TR ISSN: 0038-0407

This is a new faculty member!

But these publications still count (2001-2006)

Books

• Counted only if they are cited in the “leading journals”

• Journal titles are handled systematically There are fixed abbreviations for them

• Book titles aren’t!

• Inconsistent in how books are cited in different journals

Any Part of a Book’s Title Might Be Chosen

• The actual bibliographic reference (full): Microcomputer applications in qualitative research. Qualitative research methods series No. 14 Sage University Paper Series. Canoga Park, CA: Sage, 1988.

• 1 MICROCOMPUTER APPL • 17 MICROCOMPUTER APPL Q    • 15 MICROCOMPUTER APPLIC • 1    MICROCOMPUTER APPLIC 14• 1 QUALITATIVE RES M • 5 QUALITATIVE RES METH • 1 QUALITATIVE RES METH (wrong year entered –

1998)• SAGE U PAPER SERIES• SAGE U PAPERS SERIES

Series titles (two!)

Various and sundry abbreviations of title

Faculty need to provide FULL titles (including series title, subtitle, extraneous verbiage near the title, etc.

At Risk Faculty

• Your name is common (Smith, Green, Johnson, Lee, etc.);

• Your name, as it appears in your published work, is not spelled or abbreviated consistently;

• Your name has changed (e.g., due to marriage if your current name is different than your maiden name);

• You were on leave and, while working at another university or a research institute temporarily, you provided publishers with non-WSU contact information;

• You published a paper while working temporarily in a foreign country and gave your publisher a foreign postal code.

• You published a book in the past five years (2001-2006).

You may be at risk of a citation undercount of up to 40% if:

Recommended Preparation Procedure For All Faculty

1. Examine offprints or online full-text reproductions of all your publications that have appeared within the past five calendar years (2001-2006). Those in the Arts and Humanities should examine the last ten calendar years (1996-2006).

2. Examine the affiliation information on each of your publications.

3. If there are any inconsistencies in the way your name is printed (including whether your middle initial is present or absent), note them.

4. Make a list of all the zip codes that the publisher has actually printed with your affiliation information.

5. Update your c.v. (you will be asked to submit it)

November 27, 2006

Howard Grimes

NRC Doctoral Program

Assessment