Tier 1 and Tier 2 Programs – Assessment Report

Department or program name: Molecular BiologyDegree/program assessed: Molecular Biology Undergraduate ProgramSubmitted by: Rachel Watson/Mark StaytonDate submitted: October 1, 2011

What is your research question about student learning?

This assessment project, the Knowledge Survey Project, assesses certain sub-sections of each of the three major learning outcomes for the department (as more fully described under the next question). It was conducted in MOLB 2210/MICRO 2210, General Microbiology, a core foundational course required of all majors. This project attempted to answer the following research questions:

1) What level of knowledge and confidence exists for student in General Microbiology at the beginning and end of the semester?

2) Is there a relationship between confidence and actual knowledge?3) Are knowledge surveys a useful tool for assessing student learning in the

sciences?

What program or department-level student learning outcomes were assessed by this project?

The Molecular Biology Department has adopted three learning outcomes:

1) Graduates will be able to propose hypotheses that explain novel biological phenomena;

2) Graduates will be able to propose experimental tests of hypotheses; and 3) Graduates will be able to effectively communicate the significance of classic

biological principles and emerging biological developments to both expert and lay audiences.

Describe your assessment project and provide pertinent background information.

In General Microbiology, during the spring semester of 2011 and continuing into this fall semester of 2011, knowledge surveys were pilot tested (Nuhfer & Knipp, 2003). These surveys present a series of questions that can be skill- or content-based, that unlike traditional pre-/post-tests, do not ask students to answer the questions, but instead ask them to rate their confidence in answering the questions. These surveys have touted advantages for encouraging student and instructor preparation as well as tracking missing concepts in curriculum (Nuhfer & Knipp, 2003).

1

The Knowledge Survey Project began in late 2009 with the development of a list of concepts of which molecular biology majors are expected to have knowledge upon graduating. Pursuant to this has been the development of a large database of questions (many appropriate to both Molecular and Microbiology majors) that have been coded into Bloom’s six levels of taxonomy (Bloom, 1956) (Figure 1 below). This was exploratory research with a two-fold goal: 1) to determine if knowledge surveys would have value for our department/program and 2) to determine whether there would be a positive correlation between class confidence and class correctness as measured using a combined knowledge survey and pre- / post- test.

Questions on the survey were coded into Bloom’s six levels of taxonomy and thus include recall, comprehension, application, analytical, synthesis and evaluation questions (Bloom, 1956). The second and third sub-sections of learning outcome 1 are, “Appropriate application of fundamental principles” and “Sub-discipline specific application of facts and principles”. These are assessed by application questions on the knowledge survey. Evaluative knowledge survey questions assess learning outcome 1, sub-point #2, “Consideration of alternative hypotheses.” Synthesis questions on the knowledge survey assess learning outcome 2 as they ask students to produce an original idea, propose an experiment or design a construct. Although learning outcome 3 is less heavily assessed, comprehension questions ask students to interpret scientific literature and communicate biological principles.

Forty questions from the database were selected and used to generate a pre-test / knowledge survey and post-test / knowledge survey that were administered on a secure classroom management system (eCompanion). Students were asked to answer each question and to rate their corresponding confidence in their answer for each question. Students answered A if they were confident that their answer was sufficient for grading purposes; they answered B if they felt they could answer at least 50% of the question or could research the answer in 20 minutes or less; they answered C if they felt unconfident that they could answer the question for grading purposes. The pre-test and knowledge survey was administered on the first week of class (January 10-14, 2011). The post-test and knowledge survey was administered near the completion of the course (April 20- 27, 2011). Data for those students signing informed consent were analyzed (N = 70).

Provide relevant data to answer your research question. What are the key findings?

Scatter plot trend analysis of correctness versus confidence for both pre- and post-test data show a strong, positive correlation. When pre-test correctness is plotted as a function of pre-test confidence, a correlation coefficient (Pearson’s R) of 0.70 is found. The same analysis for post-test data shows an even stronger positive correlation (Pearson’s R = 0.85). These data are summarized in data graph 1. To further verify the relationship between confidence and correctness, the mean correctness for all students answering either A, B or C on the confidence survey was determined. Descriptive statistics were performed using an ANOVA with alpha set to 0.05. Findings show that for students

2

answering A on the pre-test confidence survey, the mean correctness was 73% (standard error of ±1.7%). Those students answering B on the pre-test confidence survey had a mean correctness of 42% (±1.1%). Finally, a mean correctness of 3.3% (±0.6%) was found for students answering C on the pre-test confidence survey. The same analysis done for post-test data showed similar trends. Those who answered with an A-level confidence had a mean correctness of 77 % (±1.1%), B-level confidence had a mean correctness of 54% (±1.2%), and C-level confidence had a mean correctness of 8% (±1.3%). These data are summarized in data graphs 2 and 3. Both the pre-test and post-test analyses demonstrated significant differences (p<0.005) in correctness between confidence levels.

Data were also analyzed for changes in correctness and confidence (pre- versus post-test) within the six levels of Bloom’s taxonomy (Figure 1). Data show an overall increase in both confidence and correctness in each level of Bloom’s taxonomy from pre- to post-test (Data graphs 4 & 5). Confidence and correctness is lowest in the synthesis category on both pre- and post-tests. The smallest degree of improvement for both measurements is seen in this level as well. The greatest improvement in correctness and confidence is seen at the evaluative level. Improvements in both confidence and correctness were greater at the recall and comprehension levels, with application lagging behind.

Initial research questions have thus been addressed. Findings show a positive correlation between correctness and confidence and an increase in both measures from pre to post test. There was also general usefulness of this method to the Molecular and Microbiology programs. Because the pilot study was performed in General Microbiology, a traditionally sophomore-level course, the greater improvements in recall and comprehension are expected. However, the outstanding increase in confidence and correctness in Bloom’s highest level (evaluation) is very positive and worthy of note. On a less optimistic note, synthesis and application skills clearly need to be cultivated and thus changes in instructional emphasis relating to these levels are already being considered. A second pilot project is underway in General Microbiology this semester. Changes include the replacement of the A, B, C confidence scale with a 1 through 5 numeric scale as well as clarifications in certain questions and instructions.

The strong, positive relationship between correctness and confidence is a key finding. This finding provides tentative data supporting the use of confidence surveys in their traditional form in which students do not answer the questions, but instead simply rate their confidence. However, further pilot testing is needed to assure that even when students are not simultaneously asked to answer the question, this positive correlation is still seen.

Equally important findings include the changes in confidence and correctness from pre- to post-test in Bloom’s six levels of taxonomy. First and foremost, students showed an increase in both confidence and correctness on every level. Students seem to display the lowest confidence and correctness in the synthesis category. At this level and at that of analytical and application, even on the post-test, the number of students relating lack of confidence still outnumbers those with the greatest confidence. On an opposite note, the

3

increase in both confidence and correctness at Bloom’s highest level (evaluation) shows a clear strength. Thus, some weaknesses and strengths in instruction have been identified and instructional goals can now be evolved to address both.

Describe the meaning of your results as they relate to the program strengths and challenges. What changes to the program or curriculum have been made, are planned, or contemplated in the future as a result from this assessment project?

In the General Microbiology class, as mentioned above, the pilot data are already being considered to make changes in instructional emphasis relating to Bloom’s taxonomy. A second pilot project is underway in General Microbiology this semester to fine tune the knowledge survey and the process itself. Overall, the response from other faculty within Molecular Biology and Microbiology has been positive. It is hoped that with increasing pilot data, the Molecular Biology department will launch a programmatic knowledge survey project (across the curriculum) in the fall semester of 2012. The Microbiology program has adopted knowledge surveys in a slightly different format as a part of programmatic assessment and professors in some upper-division molecular biology classes have also begun to integrate knowledge surveys into their courses.

References:Bloom, B. S. (1956). Taxonomy of Educational Objectives- The Classification of

Educational Goals. Handbook I. - Cognitive Domain, New York, NY: David McKay.

Nuhfer, E. B and Knipp, D. (2003) The knowledge survey: a tool for all reasons. To Improve the Academy. 21, 50-78. accessed on August 17, 2009 athttp://www.isu.edu/ctl/facultydev/KnowS_files/KnowS.htm

4

Figure 1: Bloom’s Six Levels of Taxonomy

Figure created by Rachel Watson and adapted from Nuhfer (2003) as well as Overbaugh & Schultz (n. d.)

5

Data Graph 1. Scatter plot and trend analysis of correctness versus confidence

Data Graph 2. Pre-test comparison of confidence rating to correctness.

6

Data Graph 3. Post-test comparison of confidence rating to correctness.

Legend for data graphs 1 and 2: X-axis – Confidence survey answer.Y-axis – Mean correctness of answers. This is set of a scale from 0 to 1.0, with 1.0 being 100%

correct.

7

Data Graph 4. Confidence rating according to Bloom’s Taxonomy Level. Pre-test and Post-test.

Legend: X-axis – Levels in Bloom’s Taxonomy, with the first bar in each category being the pre-test

confidence score and the second bar being the post-test confidence score.Bloom’s Taxonomy Levels:1) Recall2) Comprehension3) Application4) Analytical5) Synthesis6) Evaluation

Y-axis – Mean confidence score with 5 being equivalent to a confidence rating of A.

8

Data Graph 5. Correctness according to Bloom’s Taxonomy Level. Pre-test and Post-test.

Legend: X-axis – Levels in Bloom’s Taxonomy, with the first bar in each category being the pre-test

correctness score and the second bar being the post-test correctness score.Bloom’s Taxonomy Levels:1) Recall2) Comprehension3) Application4) Analytical5) Synthesis6) Evaluation

Y-axis – Mean correctness of answers. This is set of a scale from 0 to 1.0, with 1.0 being 100% correct.

9