food and agricultural education information system (faeis) project user opinion survey 2005
DESCRIPTION
Food and Agricultural Education Information System (FAEIS) Project User Opinion Survey 2005. Thursday, June 23, 2005 Washington, D.C. H. Dean Sutphin Yasamin Miller ProfessorDirector, SRI Agriculture & Extension EducationCornell University Virginia Tech [email protected] - PowerPoint PPT PresentationTRANSCRIPT
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Thursday, June 23, 2005Washington, D.C.
H. Dean Sutphin Yasamin MillerProfessor Director, SRI
Agriculture & Extension Education Cornell UniversityVirginia Tech University [email protected]
[email protected] www.sri.cornell.edu
Food and Agricultural EducationFood and Agricultural EducationInformation System (FAEIS) ProjectInformation System (FAEIS) Project
User Opinion Survey 2005User Opinion Survey 2005
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
• To understand from the FAEIS users, their experiences with the system and its usefulness
• To gather input from FAEIS users in higher education on improvements and changes that are needed in the future
Purpose
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
1. Solicit input about the user friendliness of FAEIS
2. How well is FAEIS meeting its goal of supporting higher education in food, agriculture, and natural resources sciences
3. How well is FAEIS meeting its goals to support the national network of universities
4. Determine the comprehensive view of current and future use of FAEIS as reported by feedback from expert in the field
Objectives
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
5. Determine the effectiveness of FAEIS and future direction in terms of context, input, process, and product
5.1 CONTEXT
Includes the elements of the university system in which FAEIS is operating including alignment with needs of the national network of universities, reliability of FAEIS, and interactions that university personnel have with FAEIS
Objectives
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
5.2 INPUT
Asks the user to consider the human, fiscal, and other resources required for the FAEIS to operate within the university system such as commitment of staff time and variables related to data entry.
Objectives
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
5.3 PROCESS• The user is asked to evaluate process variables
(i.e. timing of data entry and help desk activity) required for development, maintenance, and continuous improvement.
Objectives
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
5.4 PRODUCT• Characteristics of project outcomes and uses of
the system such as quality of report generators, reliability of the system components, and need for further refinements are evaluated.
Objectives
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Methodology• Survey Research using Case Study
Methodology
• Administer Survey Instrument to Population of FAEIS User Group
• Focus Group using Expert Panel
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Structure of evaluation distributed to FAEIS panel of experts Fall 04 for review and reaction Review and refinement of structure based on input Decision to delay evaluation until Spring 05. The survey instrument was developed by SRI in collaboration with the FAEIS central project staff and USDA Higher Education administration responsible for the project.
Methodology
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Population and Sample
1862 Land-Grant7791890 Land-Grant1351994 Land-Grant26Non-Land-Grant 419Other (US, USDA, Priv, Org) 57
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
1. Outlining overall survey objectives
2. Development of survey dimensions
3. Development of items within dimensions
4. Internal testing-administration of survey to individuals on the FAEIS administrative team
5. Revision of instrument based upon internal testing
Instrument Development
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
6. Pilot test - via web• N=35 individuals
10 pre-selected due to qualifications
25 randomly selected• Personalized invitation e-mails sent with unique
URL links on 5/26/05 • Reminder e-mail sent to non-respondents on
5/31/05• Total: 17/35 completions• Response rate: 48.6%
Instrument Development
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
7. Revision of questionnaire based upon pilot results• Panel of experts used to determine instrument
validity• Variance in responses• Not all questions applicable to all respondent types• Additional response categories added, few minor
changes made• Conclusion: revised instrument was valid and
reliable and able to capture the desired information
Instrument Development
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
8. Launching of tested instrument• Personalized invitation e-mail with unique URL
link sent on June 6, 2005
Data Collection
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Number of Responses by Institution TypeJune 17
Institution Type Total Completions Refusals Response Rate
1862 Land-Grant 779 216 28 27.7%
1890 Land-Grant 135 25 3 18.5%
1994 Land-Grant 26 4 0 15.4%
Non-Land-Grant 419 99 10 23.6%
Other (US, USDA, Priv, Org) 57 8 3 14.0%
TOTAL 1,416 352 44 24.8%
Response Rate
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
• Send up to five reminder e-mails to non-respondents
• Identify institution types with low response rates and follow-up with reminder phone calls
Follow Up of Non Respondents
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Control for Non Repondent Error
• Comparison of non respondent to respondents data
• Determine reason for lack of participation in survey
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Indepth Analysis and Probing
• Use FAEIS Expert Panel and Focus Group
• Data Transcribed and Reported Internally
• Anonymity of Response
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
• Analyses to portray variables and comparison on dimensions of particular interest
• Conclusions
• Recommendations
• Executive Summary
Final Report
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Who are the respondents:
352 respondents out of 1416 by June 17
• 82.4% use the FAEIS system and enter data• 4.4% do not enter data, but use the system• 13.2% do not use the system at all
(of these significantly more NLG (20.2%) and Other (62.5%) are non-users)
Preliminary Findings
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Q: Can you obtain student placement data? (significantly more NLG (30%) cannot get student placement data)
Q: Are we collecting data from the institutions which you consider your peers?
Q: Do a sufficient number of institutions report data to allow for meaningful and helpful analysis?
Student Placement Data
52.6
36.0
17.65.1
7.4
29.8
42.5
56.6
52.4
0
10
20
30
40
50
60
70
80
90
100
Can obtain student placementdata
Collecting data from peerinstitutions
Sufficient number of institutionsreport
Per
cen
tag
e o
f re
spo
nd
ents
Do not know
No
Yes
Preliminary Findings: Context
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Q: How would you rate your ability to obtain the following data in the format required by FAEIS:
Ratings of Ability to Obtain Data in the Format Required by FAEIS
9.13.5
31.029.6
11.4
26.628.7
21.9
7.9 8.7
23.2
11.4
21.0 21.728.5
11.812.8
32.7
23.5
6.6
2.21.71.3
23.0
0
10
20
30
40
50
60
70
80
90
100
Faculty discipline by rankby gender by ethnicity
Degrees awarded by levelof study by gender by
ethnicity
Enrollments by level ofstudy by gender by
ethnicity
Placement by level ofstudy into various sectors
Per
cen
tag
e o
f re
spo
nd
ents
Very easy Easy Neither difficult nor easy Difficult Very difficult Do not know
Preliminary Findings: Input
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Q: Are the instructions in the FAEIS system clear?
Clarity of Instructionsin the FAEIS System
Yes, instructions
are clear83.0%
No, instructions
are not clear
17.0%
Preliminary Findings: Process
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Q: Is the support offered by the Help Desk adequate?
Whether Support Offered byHelp Desk is Adequate
(among those who have contacted Help Desk)
Yes95.6%
No4.4%
Preliminary Findings: Process (continued)
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Q: How useful is the information collected in FAEIS:Ratings of Usefulness of Different Types of Information
Collected in FAEIS
19.7 18.614.5 10.8
26.3
30.5 30.0 27.9
26.225.7
4.4 8.0 9.3 7.3
8.98.7
3.6 3.6 3.63.2
6.9
4.1
37.1 38.2 38.2 38.5 41.3 43.550.6
29.5 27.520.2
28.3
2.82.4
0
10
20
30
40
50
60
70
80
90
100
Degreesawarded
Enrollment Ethnicity Gender Faculty Placement ofgraduates
CIP
Per
cen
tag
e o
f re
spo
nd
ents
Very useful Somewhat useful Not very useful Not useful at all Do not know
Preliminary Findings: Product
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Q: To what extend does FAEIS add value to other information sources that are available?
Value-Added of FAEIS Over Other Data Sources
11.6
8.010.4
32.5
37.3
0
10
20
30
40
50
High degree Moderate degree Low degree None Do not use orsearch otherdatabases
Per
cen
tag
e o
f re
spo
nd
ents
Preliminary Findings: Product (continued)
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
• 84.8% do not use Custom Report Builder
Preliminary Findings: Product (continued)
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Q: Who are users of FAEIS data at your institution?
Users of FAEIS at Institution(among those using data; multiple users allowed)
36.0
1.5
24.1
62.1
4.4
0 10 20 30 40 50 60 70 80 90 100
Do not know
Other
Government officials
Faculty/Researchers
Universityadministrators
Percentage of respondents
Preliminary Findings: Product (continued)
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Q: In general, to what extent is it useful for your institution to participate in the FAEIS database?
Usefulness of Institution's Participation in FAEIS Database
16.3
8.3
4.2
28.8
42.4
0
10
20
30
40
50
Very useful Somewhat useful Not very useful Not useful at all Do not know
Per
cen
tag
e o
f re
spo
nd
ents
Preliminary Findings: Product (continued)
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Preliminary Thoughts on Conclusions and Recommendations
• Need further analyses to explain the high rate of “don’t know”. Is this response isolated to a particular population? How can we correct this lack of information?
• Preliminary data show a positive view of FAEIS on most variables• Need to determine why it is more difficult for non land grants to
access data• Custom report builder is in early stages of release and current data
may not reflect full potential
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Preliminary Thoughts on Conclusions and Recommendations
• FAEIS seems to be satisfactory in terms of comprehensive elements involving context, input, process, and product
• Respondents were not forthcoming with significant new and innovative future development recommendations
• FAEIS seems to be meeting its goals in terms of user friendly and meeting needs of the university community.
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Final Report Format
• Are there recommendations for the format, style or types of analyses?
• Are there recommendations on developing the web report or the written report?
• Any suggestions on dissemination of findings?
FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey
20052005
Your thoughts – opinions?
Questions