1 a longitudinal survey program to meet institutional and academic department assessment needs bruce...
Post on 20-Dec-2015
214 views
TRANSCRIPT
1
A Longitudinal Survey Program to Meet Institutional and Academic Department
Assessment Needs
Bruce P. [email protected]
AIRPO Annual ConferenceWest Point, New York June 16-18, 2004
2
Albany History and Forward-looking Needs:
Long tradition in assessing general student outcomes•W.K. Kellogg Foundation support•Albany outcomes assessment model•Ad hoc studies•Albany assessment report series http://www.albany.edu/ir/reports.htm
Future-oriented developments•Middle States accreditation requirements•Campus assessment plan•Building a campus culture that embraces assessment at the unit level as well as at the institutional level
3
Personal Traits
Pre-CollegeCharacteristics
Parents’ Education
Aptitude TestScores
High SchoolAchievement
CollegePreparation
Involvement
Receptivity to ServicesEducational Aspirations
Age, Ethnicity,& Gender
College Experiences
Acad Integration
Social Integration
Acad ConscientiousnessClassroom Experiences
Faculty ContactField of Study
Peer RelationsEmployment
Extra-Curr. ActivitiesResidential Experience
Educational Outcomes
Arts & Letters
Disciplinary Study
Scientific Method
Academic Achievement
Persistence/Graduation
Alumni Outcomes
Graduate DegreesEarned
OccupationalStatus
Income Level
Satisfaction withJob and Career
Leadership& Service
Awards& Recognition
Personal
Interpersonal Skills
Openness & Tolerance
FunctioningIndependently
Parental Influence
Inst’l Integration
Inst’l CommitmentAdvisement
Financial AidAffinity of Values
AlumniGiving
Academic
Acad Motivation: Intrinsic, Extrinsic
Self-regulation
The Albany Outcomes Assessment Model
4
The Albany Longitudinal Survey Program
Year 1 Year 2 Year 3Year 4+
Entering Student Senior ExperienceSurvey Survey
Freshmen StudentExperience Survey
Surveys administered every year
5
Entering Student Survey
•Focus on student pre-college characteristics to identify students at risk prior to first semester of study
•Concentrate institutional resources on students in-need of attention - individual student profiles
•Communicate general findings back to new freshmen to raise awareness of successful student profiles
•One leg of the Advisement Services Center’s tripartite assessment program
•http://web.albany.edu/ir/samplesurveys/ess.html
6
7
8
9
10
Parental InfluenceComposite of mother's and father's educationMy parents encourage me to do the best in whatever I doI can go to my parents when I have a problem
Educational Aspirations (scale: importance)To gain a broad, well-rounded education (in the sciences, humanities, and arts)To learn to think creatively and analyticallyTo learn more about myself, my values, and my life's goalsTo learn more about other cultures and groups of people.
Involvement (scale: degree of chance)Participate in intercollegiate or intramural sportsBelong to a fraternity or sororityJoin a student club (academic, political, or cultural)Participate in student government, student newspaper, and/or campus radio station
Receptivity to Services (scale: degree of chance)Visit professors during their office hoursAttend a study group from one of your classesContact my academic advisor more than once per semesterVisit the Career Development Center
College Preparation (scale: yes or no)Pre-calculus or calculsForeign languageScienceAdvanced placement (AP) courses
Pre-college Scales
11
Pre-college Scales – cont’dAcademic Motivation - Intrinsic (scale: not true to true)
I enjoy the classes that challenge my mind.I will feel successful if I have the highest test scores.Thinking about my future motivates me to work hard in school.Solving difficult problems in my classes gives me a sense of satisfaction.I feel successful when I learn something interesting.Learning new things gives me a sense of accomplishment.Schoolwork is interesting because you see yourself making progress.I read books not assigned in classes.
Academic Motivation - Extrinsic (scale: not true to true)I study hard because I want to prove myself as capable as anyone else in class.I worry about not doing as well academically as others in college.I don't feel I can gain much in college other than obtaining a degree.I feel that as long as my teachers are satisfied with my academic performance, that is
enough.I don't really care how well I will do in college, as long as I can pass.Getting a high paying job is the main reason I study hard in college.I feel ashamed if I don't do as well as other students do in college.I worry about failing exams and getting into academic trouble.
Self-Regulation (scale: not true to true)When I get something wrong in school, I always stop and try to find out what went wrong
rather than simply asking my instructor the correct answer.I take notes while reading textbooks.When I run into difficulties in schoolwork, I feel if I put in extra effort, I will eventually figure them out.When I am reading my textbooks, I stop and check whether I really understand what I have
just read.If I study harder my academic abilities will improve.I keep a regular study schedule.When learning a new concept, I never check if I really understand its meaning.
12
Entering Student Survey
•Focus on student pre-college characteristics to identify students at risk prior to first semester of study
•Concentrate institutional resources on students in-need of attention -individual student profiles
•Communicate general findings back to new freshmen to raise awareness of successful student profiles
•One leg of the Advisement Services Center’s tripartite assessment program
•http://web.albany.edu/ir/samplesurveys/ess.html
13
Freshmen Student Experience Survey
•Examine changes in students’ psychological make-up after the first college year
•Gauge institutional engagement and relationships with educational outcomes
•First pass at gauging centralized student advisement
•Communicate general findings back to students and faculty to support the development of a campus culture of assessment
•http://web.albany.edu/ir/samplesurveys/froshses.html
14
15
16
17
18
19
20
21
22
23
24
25
College Experience Scales & Items
Academic Conscientiousness (scale: various 5 pt scales)How frequently have you worked harder than you thought you could to meet an
instructor's expectations?Developed your study skills.Utilized your study skills.I do very little studying on weekends.I put a good deal of effort into being well prepared for examinations. My primary goal at UAlbany is to take advantage of academic opportunities.Doing homework and studying outside of class.
Classroom Experiences (scale: never to almost always)Been satisfied with your academic experiences?Had out-of-class assignments that were good learning experiences?Worked with other students on class assignments?Received feedback (written or oral) from instructors on the quality of your work?Had to critically appraise the value of information, arguments, or methods in your classes
or class assignments?
Faculty Contact ItemsHow frequently have you had discussions, meetings, or conversations with instructors
outside of class?
Visit professors during their office hours?Seek out a faculty mentor?
Academic Integration:
26
Social Integration:
College Experience Scales & Items– cont’d
Peer Relations (scale: disagreement to agreement)I have developed strong friendships with other students. My interpersonal relationships with other students have had a positive
influence on my personal growth.The student friendships I have developed have been personally satisfying.It has been difficult for me to make friends with other students.I know several UAlbany students who would be willing to help me if I had a
personal problem.My interpersonal relationships with other students have had a positive
influence on my intellectual growth.I am confident in my ability to make friendships.
Extra-curricular activities (scale: yes or no)Participate in intercollegiate or intramural sports?.Belong to a fraternity or sorority?
EmploymentWorking on-campus (volunteer or paid).Working off-campus (volunteer or paid).
27
Institutional Commitment (scale: yes or no)Visit the Writing Center?Visit the Career Development Center?Contact your academic advisor more than once per semester?
Financial Aid
Advisement Items (scale: disagreement to agreement)Advisor was knowledgeable about course requirements for the major(s). Advisor was knowledgeable about course requirements for the minor(s). Advisor was helpful in assisting with scheduling/ registration procedures. Advisor showed genuine interest in my academic progress.Advisor went over my degree audit with me to inform me of course requirements I
have not yet met.Advisor was knowledgeable about academic policies (e.g., graduation
requirements, GPA requirements, General Education guidelines, residence requirements, etc.).
Advisor was available when I needed him/her.Advisor showed concern for my personal growth and development.Advisor helped me to identify career areas (or graduate school opportunities)
which fit my skills, abilities, and interestsAdvisor encouraged me to talk about myself and my college experience.Advisor was easy to talk with. Advisor promptly returned my phone calls, e-mails, and/or messages.
Institutional Integration:
College Experience Scales & Items– cont’d
28
Arts & LettersWriting effectively.Developing intellectual curiosity.Developing the ability to formulate creative ideas and/or solutions.Appreciating artistic and creative expression.Placing current problems in historical perspective.
Scientific MethodEvaluating ideas, materials, and methods critically.Developing problem solving skills.Thinking analytically and logically.Understanding scientific findings.Understanding mathematical concepts.
Disciplinary StudyGaining factual knowledge.Learning how to learn.Synthesizing a body of information.Understanding a particular discipline's research methods.Understanding a particular discipline's various schools of thought.Understanding the inter-relatedness of different fields of study.Using information and/or computer technology in your academic discipline
Educational Outcomes
Academic (scales: none to very large college contribution)
29
Educational Outcomes – cont’d
Social (scales: none to very large college contribution)
Interpersonal SkillsSpeaking effectively. Developing interpersonal and social skills.Functioning effectively as a member of a team.Preparing for active participation in a democratic society.Coping with conflict.Adapting to different social situations.
Openness & ToleranceUnderstanding cultural differences.Gaining exposure to a variety of new intellectual areas.Developing an openness to new ideas.Coping with moral and ethical issues.Developing a better understanding of myself (e.g., interests, talents, values,
limitations).Understanding cultural differences.Relating well to people of different races
Functioning IndependentlyFunctioning independently (self-reliance).Exercising personal responsibility.Developing a better understanding of myself.
30
Freshmen Student Experience Survey
•Examine changes in students’ psychological make-up after the first college year
•Gauge institutional engagement and relationships with educational outcomes
•First pass at gauging centralized student advisement
•Communicate general findings back to students and faculty to support the development of a campus culture of assessment
•http://web.albany.edu/ir/samplesurveys/froshses.html
31
Senior Survey
•Examine changes in students’ psychological make-up over their UAlbany career
•Gauge institutional engagement and relationships with educational outcomes
•Connect engagement and educational outcomes with student experiences in the major
•Focus on academic advisement in the major
•Communicate general findings back to students, faculty, and academic units to support the development of a campus culture of assessment
•http://web.albany.edu/ir/samplesurveys/seniorses.html
32
33
34
35
36
Personal Traits
Pre-CollegeCharacteristics
Parents’ Education
Aptitude TestScores
High SchoolAchievement
CollegePreparation
Involvement
Receptivity to Services
Educational
Aspirations
Age, Ethnicity,& Gender
College Experiences
Acad Integration
Social Integration
Acad ConscientiousnessClassroom Experiences
Faculty ContactField of Study
Peer RelationsEmployment
Extra-Curr. ActivitiesResidential Experience
Educational Outcomes
Arts & Letters
Disciplinary Study
Scientific Method
Academic Achievement
Persistence/Graduation
Alumni Outcomes
Graduate DegreesEarned
OccupationalStatus
Income Level
Satisfaction withJob and Career
Leadership& Service
Awards& Recognition
Personal
Interpersonal Skills
Openness & Tolerance
FunctioningIndependently
Parental Influence
Inst’l Integration
Inst’l CommitmentAdvisement
Financial AidAffinity of Values
AlumniGiving
Academic
Acad Motivation: Intrinsic, Extrinsic
Self-regulation
Entering Student SurveyID at-risk studentsGeneral profiles of successful students
Freshmen & Senior SurveysExpansion of the Albany ModelGauge institutional engagementEvaluate advisement (centralized & in the major)Build culture of assessment (generally & in acad units)
37
Pros
•Longitudinal survey program tailored to institutional needs
•Demonstrates institutional commitment to self-examination
•Amenable to post-hoc analyses of special student populations
•Immediate impact in enhancing student retention
•Promotes academic unit use and participation
•Cost effective approach
Cons
•No benchmarking •Subject to validation and reliability criticisms
•Considerable start-up investment by IR staff and collaborators
•Requires IT infrastructure and support
38
END