665 session11-data analysis-s13

Student Learning

Outcomes:

Data Analysis &

Interpretation

Student Learning Gains and Input for Revisions to Instructional Design

•Dr. Diane Nahl Spring 2013

•LIS 665 Teaching Information Technology LiteracyUniversity of Hawaii LIS Program

Instructional Design Stages

① Needs Assessment: What is and what is needed?

② Define Goals & Objectives: What will they do and learn?

③ Select Formats, Methods & Materials: How is best?

④ Devise Test & Evaluation Procedures: Prototyping it.

⑤ Construct & Teach Prototype: Performing it.

⑥ Evaluate & Analyze Outcomes: What did they get?

⑦ Revise & Recycle Steps: What will be different next time?

Nahl 2013LIS 665 Teaching Information Technology Literacy 2

Blended Librarians• Embedded librarianship (2004 Barbara

Dewey)

• Online librarianship

• Cybrarian

• Collaboration and partnerships with teaching faculty, students, and institutional or community events and initiatives.

• Often immersed within ICT environments

• Long-term involvementNahl 2013LIS 665 Teaching Information Technology Literacy 3

Learning Fundamentals

• Learning is about actively making connections:

• Biological: establishing neural networks, activating emotion receptors, and developing muscle memory

• Conceptual: connecting ideas, making meaning, and problem solving

• Experiential: interacting with self, others, and the socio-technical information environment


Learning Fundamentals

• Learning is a developmental, cumulative process involving the whole person.

• Learning is shared within a social process.

• Learning that is constructivist, active and immersive is longer lasting.

• Learning requires frequent feedback.

• Learning often takes longer than teaching.

• Learning is sometimes instantaneous.


Holistic Instructional Design for the Three-fold

Self• Teach the Heart

• [A] Affective learning

• Teach the Mind• [C] Cognitive learning

• Teach the Body• [S] Sensorimotor learning


Worksheet Data Analysis

• Radcliff et al. Ch 11: Which types of scoring guides will you use on your outcomes data that measured performance?

• In your case, forms, worksheets, and discussion posts were used to gather responses in active learning exercises.• How will you score the responses on the worksheets? How

will you assign levels of completeness or correctness?

• To answer the questions above complete the data analysis exercise Steps 1 through 8 on the following slides by April 4 (2 weeks).



① First Step: Make a copy and then examine one of the online worksheets from an activity and read all of the responses from every student or team/group on that worksheet.

a. Assign an ID code to each student directly on the worksheet and use that ID code in the spreadsheet. (Work only on copies of student worksheets)

② Second Step: Set up your scoring criteria for correct, partial, and incorrect or blank responses.

a. You may score multi-part questions separately (e.g., Q1a, Q1b, etc.

b. You may use various approaches given in this chapter but you must assign numerical scores for each level of correctness or ideal to each worksheet response/answer.



③ Third Step: Read each worksheet again and score the individual question item responses. Score all of the worksheets for that exercise.

④ Fourth Step: Use the spreadsheet (shared last week) to record the individual question scores (so you know how everyone did on each question as in the Hillyer article Fig. 7.2).

a. This requires an ID column and showing each student’s data in a row (e.g., Student 1, Student 2, etc.) and each question or part of a question in columns, then enter the student scores in the appropriate column.

b. For clarity add a meaningful content word or phrase to the question number (e.g., Q4 Search Terms; Q5 Boolean) on the spreadsheet to easily show what the scores relate.



⑤ Fifth Step: Sum the columns and compute the average score for each question. [total of all scores on that question/total number of students]

⑥ Sixth Step: Sum the rows and compute the average for each student and convert to a percentage.

a. [(total of all scores on every question by each student/maximum possible score) x 100]

⑦ Seventh Step: Sum the total scores for all students to get a global score for the entire class on that worksheet, compute the average and convert to a percentage.

a. [(global total of all questions for all students/maximum possible score) x 100]



⑧ Eighth Step: Examine the spreadsheet and look for patterns, e.g.,

a. What is the range of scores on each question (highest and lowest scores)?

b. Which question was most difficult for most students? Which question was easiest for most students?

c. If you graded each student based on the maximum possible, what letter grade would each receive (90-100% A; 80-89% B; 70-79% C; 60-69% D; 50-59% F)? What letter grade would the entire class receive on this worksheet?

d. What other patterns do you see in your data? These findings will be part of the Assessment Report write-up.



Data Analysis Exercise: Content Analysis Method Ch 13

Radcliff et al.

• Examine open-ended data from your session by reading all of the responses, may use Psych GG comments posted within 2 weeks after the session. Follow Content Analysis steps 1-7 + 8:• Select an open-ended narrative data instrument (minute

writing reflection exercise, open-ended worksheet responses or session evaluation, or other type) and read the entire set of responses.

• Begin with Step 1 Determine What Questions You Want to Answer and articulate what you intend to analyze and why.

• In Step 2 determine what content you will extract and count for the analysis.

• In Step 3 define what you will count as an occurrence• Necessary to be very specific and concrete to avoid ambiguity


Data Analysis Exercise:

Content Analysis Method• In Step 4 Create a Coding Sheet to organize

data extraction (Google document spreadsheet)

• In Step 5 Pre-test the coding sheet. Each Team member codes the same section individually, then compare your work• If necessary, discuss the differences and

standardize the coding definitions to obtain nearly complete agreement in coding

• Agreement among raters should be 95% or higher



Content Analysis Method• In Step 6 after achieving high agreement in coding,

take separate portions of narrative and code it individually. • Compare coding at intervals to ensure agreement

remains high• In Step 7 Use the spreadsheet to enter and then

analyze (compute or summarize) the coded data. • It is best if you code independently and maintain

individual spreadsheets, or separate sheets in one workbook.

• Compare your individual results and compute the percent overlap (>94%)

• Repeat steps 1-7 to analyze the rest of your open-ended data as well as the PSYCH GG comments posted after your session


Content Analysis Method• In Step 8 (not numbered in text) you will turn your

spreadsheet data into information• Compute descriptive statistics of totals (frequencies),

averages (the mean), and percentages• Clusters of terms within a concept, categories and

their members• Compute individual student statistics and clusters

(encouraged for this project)• Compute inferential statistics about the population

(not required for this project)• Make a table or graphic chart to display results

together.• Give the table a meaningful title.


In Two Weeks

• Instruction Unit Parts I & II due April 4

• Guest presentation: Sean Thibadeaux, Reference and instruction librarian, Hawaii Pacific University

• Ch 8

• CTSB report

• Mackey & Jacobson

• Bring numerical data in tables and/or graphic charts from your session for data analysis


665 session11-data analysis-s13

Education

content analysis method

data analysis exercise

learning fundamentals learning

worksheet data analysis

ended data

letter grade

coding sheet

id code