Comparing computer technology skills usage and development in higher education humanities

and sciences syllabi – a research proposal using quantitative content analysis

Jeremy L. McLaughlin

January 2015

1

Amidst the changing landscape of the digital age, it is widely accepted that most

disciplines in the humanities generally fall behind the sciences in the use and application of

computer and digital technologies to the research process and in day-to-day academic activities.

There has been extensive investigation into the availability of digital resources for humanists,

their use, and possible users, pointing to a great deal of potential growth in this area (Harley et

al., 2006). While the general adoption and adaptation of technology is improving there (Dahl,

2013) without a quantitative or shared networking focus similar to the sciences, or a framework

that describes the underlying methods and functions of technology usage in the humanities, the

process is significantly more challenging (Anderson, Blanke, & Dunn, 2010; Blanke, Hedges, &

Dunn, 2009).

Much of the research in this area falls into two categories. The first deals with the

application of eScience principles to the humanities. Especially in the U.K. and Australia this

deals with technology usage within the context of advanced networking, high-performance

computing, distributed research programs, big data analytics, and visualization. The second deals

with research traditions, discipline culture, information seeking behavior, and technology use in

the humanities as enhancement to current practices. While there is overlap, the focus in the latter

is on lower-level (Kircz, 2004), common uses and applications of technology across

professionals and students in the humanities. For both, it is widely agreed that exposure to

computer technology, promotion of technology skills usage, and increased education about

technology applications outside of course-specific contexts is needed.

The lower-level application of technology skills in undergraduate and graduate courses is

the focus of this research proposal. Given the differences in the use of digital technology

between disciplines, and the need to build technology skills usage through education and

2

engagement in the humanities, this research proposal explores the main research question, “how

is the use of computer technology skills by students to complete assignments in humanities

courses different than in science courses?” This is a unique approach as the literature does not

provide examples of analysis of humanities syllabi or comparisons of computer technology skills

usage in syllabi across disciplines, institutions, and time.

While “the study of the scholarly use of technology is not rich in theory, in the sense of

empirically testable propositions that have been around long enough to be able to adequately or

explicitly solve research problems” (Zhang, 2014, p. 11) these factors may contribute to the

empirical foundations of the theoretical differences in technology adoption outside of the

sciences. This study may also relate to building a culture of technical engagement for humanities

students. Looking at how technology engagement and skills usage are integrated into academic

courses provides opportunities for more thorough applications of contemporary technology at all

levels of humanities education.

I begin by exploring the literature related to course syllabi, technical exposure/skills

development as it relates to the humanities and the research questions of this paper. Next, I will

introduce the methodology of content analysis and its specific application to technical skills

usage in higher education course syllabi. Based on measurements of the application of technical

skills to completing course assignments, our methodological discussion will continue with an

overview of syllabus coding techniques and the Syllabus Assessment Instrument. Finally, I will

provide an overview of the types of analysis and discussion I would like to draw from the data

and a possible timeline for completing this project for my MLIS thesis.

Literature review

Course syllabi and technical skills exposure/development

3

The importance of the syllabus as a tool for student-centered learning, their role in

providing details about class objectives and assignments, and their significance in administrative,

course development, and interpersonal communication domains of higher education (Eberly,

Newton, & Wiggins, 2001) is thoroughly discussed in the literature.

Please see McDonald, Siddall, Mandell, and Hughes (2010) for design implications and

use, and Fink (2011) for a detailed theoretical and empirical overview of the purpose, structure,

format, and use of course syllabi from the instructor and the student’s perspective.

The analysis of syllabi is especially important for program evaluation and assessment,

usually related to accreditation studies, curriculum reviews and changes, and studies related to

learning objectives and outcomes. Because of their important role as a communication tool and

as the only tangible and consistent representation of course objectives and assignments, syllabi

provide a standard for understanding what is being taught in courses, how that relates to other

classes in the same/other disciplines at the same/other institutions, and uncovering possible areas

for improvement.

The relationship between technology and scholars is changing, along with the definition

of technology (information technology, ICT, instructional technology) and its applications in

academia (Zhang, 2014). In the 21st century, technical skills development goes hand in hand with

information literacy and other learning objectives.

Mayo (2012) focuses on how technology is used to foster teaching and meaningful

learning through integration with objectives and assignments, as opposed to the specific software

or technology is being used. This ensures that educational objectives and the application of

technology in ways that facilitate higher learning are the driving force and not the technology

itself.

4

While knowledge transmission is the key objective, syllabi also serve the purpose of

explicit and implicit skills development for students (Eberly, Newton, & Wiggins, 2001). As

more courses are offered online and learning management systems and other forms of university

e-learning environments take shape, it is critical that students develop a core set of competencies

for effective performance (Parkes & Reading, 2013).

Teaching methods using information technology can result in higher student perceptions

of effective teaching and increased student perceptions of their own computing abilities (Wolk,

2004).

In addition to correlating with other desirable student behaviors, Wolk (2006) shows that

student computing results in higher overall student satisfaction and can be an indicator of

progress during institutional assessment and accreditation. As such, the integration of student

computing skills in all disciplines should be a priority for administration, faculty, and the library.

Technology in the humanities

In a large-scale, multiyear project designed to understand the types of digital resources

available, and their use and users in undergraduate education in the humanities, Harley et al.

(2006) provides a thorough review of the theoretical and empirical landscape.

Like many studies, they also review the hindrances or reasons for resistance to

technology adoption in the humanities. Regardless of the framework, methodology, or scope of

the research, the problem of traditions and disciplinary cultures in humanities, as they relate to

information seeking behavior and technical engagement, has remained a consistent focus in the

literature. While technology advances at a rapid pace, the sources of barriers and boundaries to

extensive adoption in the humanities have remained the same.

5

These boundaries are summarized as the particular research needs and traditions of

humanists and include: concerns around the reliance on books as a primary source of data;

perceptions of physical versus digital objects; the properties, value, and role of primary

materials; the authority and trustworthiness of digital resources; lack of a standard quantitative

research base; no reliance on networking or shared spaces; and the importance of research and

learning spaces to the research process (Anderson, Blanke, & Dunn, 2010; Kircz, 2004; Puckett,

2011; Rimmer, Warwick, Blandford, Gow, & Buchanan, 2008).

Harley et al.’s extensive 2006 study included enthusiasm as a variable in humanities

technology usage by undergraduate students. Specifically, they point out that technology use by

instructors varied by discipline but was directly related mostly to the instructor’s own enthusiasm

about technology. Enthusiasm, in turn, was associated with a complex set of variables including

instructor demographic characteristics and the institutional and disciplinary setting.

This underlying culture of research practice needs to align with technology usage and

engagement at a common level (Genoni, Merrick, & Willson, 2009). One recent example shows

that ebook usage applies to humanists and may signal a change in acceptance of digital resources

if they are applicable and discoverable for students (Dahl, 2013).

Librarians can promote information literacy through alliances with younger, digitally-

inclined students (Mayo, 2012; Puckett, 2011). In fact, when it comes to educational technology

and computer technology skills, libraries need to promote awareness and engagement with

technology at practical levels.

Kircz (2004) notes that, at that time, students in the humanities did not receive

standardized or consistent training on how to understand and apply computer technology skills in

setting outside of course-specific contexts. As such, faculty play a critical role in providing

6

direction about technology and other resources (print or digital) to be used in research and other

assignments (Head, 2008). This role is primarily carried out through the course syllabus.

Understanding the information gathering and research needs of humanists provides a

basis for the application of computer technologies and appropriate levels of technical skills

development based on aligned goals.

Research questions and hypotheses

Research question 1: How is the use of computer technology skills by students to

complete assignments in humanities courses different than in science courses?

Research question 2: What basic or advanced computer technology skills are used most

frequently in the humanities courses and in the science courses?

Research question 3: How significant is the difference and has there been a change in the

frequency of computer technology skills usage in the humanities courses over time?

Research question 4: Is there any correlation between the frequency of computer

technology skills requirements or overall technology representation in syllabi, or specific

instructor/course demographics, and variables related to humanities research preferences

(cultural differences from existing literature)?

Based on the existing literature, I hypothesize that humanities syllabi will show less frequent

explicit technical objectives and though there may a slight increase from 2002/03 to 2012/13 the

humanities will have less frequency of expected use of computer technology skills to complete

assignments than syllabi in the sciences.

Additionally, based on the makeup of the final collection of syllabi there may be some

disciplinary variance but in general I expect to see a higher frequency of print book course

7

readings (compared to digital books, or journal articles) and significantly less advanced

computer technology skills in the humanities.

Methodology

Content analysis of syllabi

Content analysis is a widely used methodology in many disciplines as it allows

researchers to reveal messages, meaning, and symbols within, and about, a communication

source. The extraction of meaning can be applied to any “text” – written, visual, or spoken

mediums of communication across print or digital spaces.

The focus of this proposal is on quantitative content analysis. White and Marsh (2006)

discuss the various applications of content analysis and provide the following specific steps for a

quantitative approach:

1. Establish hypothesis or hypotheses

2. Identify appropriate data

3. Determine sampling method and sampling unit

4. Draw sample

5. Establish data collection unit and unit of analysis

6. Establish coding scheme that allows for testing hypothesis

7. Code data

8. Check for reliability of coding and adjust coding process if necessary

9. Analyze coded data, applying appropriate statistical tests

10. Write results (White & Marsh, 2006, p. 30)

Krippendorf’s (1989) definition of content analysis focuses on “making replicable and

valid inferences from data to their context” (p. 403) while Neuman notes that “in quantitative

content analysis, you use objective and systematic counting and recording procedures to produce

a numerical description of the content in a text” (2011, p. 361).

8

The application of content analysis techniques to higher education syllabi is a popular

methodology for institutional program evaluation and assessment of the relationship between

learning objectives and learning outcomes.

Content analysis of syllabi allows for the extraction of explicit or implicit expectations of

student development in a course, and produce useful information about what faculty value in

student learning (Allen, 2004, as cited in Joe, Harmes, & Barry, 2008). As noted above, this is

especially true for questions related to course content and the characteristics of teaching and

learning practices across an institution or institutions and in specific programs and curriculum.

While course readings are the first source of information during the academic research

process for humanists (Head, 2008) no studies were found applying quantitative content analysis

to syllabi in humanities disciplines.

In an assessment of arts and humanities general education program objectives, Joe,

Harmes, and Barry (2008) use a qualitative content analysis of syllabi across disciplines at a

single institution. They analyzed thematic networks to organize trends and patterns in the syllabi

and to extract the cognitive, behavioral, and knowledge/skills components related to common

course objectives. Of the nineteen themes they develop (p. 144) one recognizes the use of “tools

and methods” and several imply the use of basic technology, but the word technology or the use

of technology as a method for inquiry in the arts and humanities is not mentioned.

Brown, Rynes, Charlier, and Hosmanek (2013) examined the correlation of institutional

variables, instructor characteristics and details of course readings (the types of books and

journals used). Similarly, McDonald, Siddall, Mandell, and Hughes (2010) carried out a multi-

method analysis of syllabi across disciplines while Wikle and Fagin (2014) were interested in the

standardization of GIS course content across institutions focusing on the use of specific

9

textbooks and software. In both studies, the content of syllabi were examined and analyzed in

similar ways. Their methods and analysis provide a meaningful framework for cross disciplinary,

multi-institutional study of course syllabi.

For this research proposal, a methodological foundation for the content analysis of

technical skills usage required by syllabi comes from Madsen, Melchert, and Whipp (2004). In

their curriculum review study, they developed a Syllabus Assessment Instrument based on the

2000 National Education Technology Standards to measure exposure to computer technology

and skills in course syllabi.

Their research is particularly relevant to the analysis of syllabi focused on a specific set

of technical skills that students are required to utilize to complete course assignments. It is also

applicable to undergraduate and graduate courses across disciplines and at any institution

(though there analysis focuses on one institution only).

The proposed methodology will examine the use and exposure of students to certain

computer technology skills. This is not the same as skills development. These tools can be

applied in a specific course and used to aid learning in other courses, regardless of expectations.

Analysis of course syllabi does not tell us about actual student engagement with the materials or

technology and they do not tell us the outcomes related to course goals and objectives.

Additionally, there is a wide possibility for students in a course to be exposed to

technologies designed to aid learning outside of the scope of the syllabus. In many cases, print

version of course syllabi may make reference to assignments and course details that are found in

the course learning management system. By design, learning management systems are supposed

to be integrated with other technologies, not least of which is authenticated direct access to

digital course content like readings, lectures/videos, etc… It is not the focus of this paper to

10

analyze computer technology skills specific to e-learning or digital course environments (Parkes

& Reading, 2013).

Additional verbal instructions or handouts may also point students to technology usage or

skills not specified or implied in the syllabus. Previous studies rely on student and faculty

interviews to assess validity of findings based on analysis of syllabi compared to participant and

instructor perceptions of course technology requirements.

Measurements and data gathering

For the purposes of this research proposal, a final source of digitized syllabi has not been

decided on. However, syllabi will be collected from two data clusters (2002/2003 and

2012/2013) and from three disciplines in the humanities (possibly history, linguistics,

philosophy) and three disciplines in the sciences (physics, biology, chemistry). Several key

decisions must be made based on the data source. For example, comparing course syllabi from a

specific discipline across institutions has been shown to be an applicable use of the content

analysis method (Wikle & Fagin, 2014) but there may be additional generalizability and

analytical benefits to using syllabi from a specified list of higher education institutions.

Madsen, Melchert, and Whipp (2004) based their Syllabus Assessment Instrument (SAI)

on technology skills standards used to prepare teachers to understand and utilize computer

technology in various contexts that support learning, facilitate higher-order thinking, and

promote collaboration and real world learning practices. Their focus is on course description,

objectives, and activities. The first two utilize latent coding and are analyzed for implicit,

explicit, or no exposure to technology skills. The latter was analyzed to determine if a targeted

set of technical skills were needed to complete course assignments. As noted above, their list of

technical skills was developed from the National Education Technology Standards.

11

Madsen, Melchert, and Whipp’s computer technology skills are based on the 2000

National Education Technology Standards. Given the timeframe of the syllabi being collected,

this skills set is appropriate as the basis for this study.

I’ve enhanced their SAI by including computer technology skills, technology resource

types, institutional variables, instructor characteristics, and additional details about course

readings from the studies using the content analysis of syllabi mentioned above. I also included

measures related to potential cultural impediments to technology use in the humanities (for

example, the presence of ebooks) as well as variables that may indicate additional technical

content or exposure not explicit in the syllabus (the availability of learning management

systems).

For a complete list of variables being analyzed in the syllabi, please see Appendix A:

Computer Technology Skills and Other Variables.

Each syllabus will be analyzed using the following procedures:

1. Demographics: year, institution, discipline, etc… Non-numeric variables will be

coded based on sample parameters (for example, each institution will have a

unique numeric code).

2. Other variables: total number of readings; total books; total journals; course LMS

= 0 (no) or 1 (yes); Print or digital = 1 or 2, respectively, as an assessment for the

entire syllabus based on whether course readings are explicitly offered in digital

format.

3. Course Description: Description will be examined for explicit, implicit, or no

mention of the use of computer technology skills in the course. Explicit = 2,

Implicit = 1, None = 0.

12

4. Course Objectives: Objectives will be individually counted and examined for

explicit, implicit, or no mention of the use of computer technology skills in the

pursuit of specific course objectives. Explicit = 2, Implicit = 1, None = 0.

5. Course Assignments and Activities: Assignments and activities will make up a

bulk of the content analysis and syllabus coding. Assignments and activities will

be examined for the explicit or implied use of technical skills to complete course

requirements. If a technical skill is present, the assignment or activity will be

coded with the corresponding technical skill number (shown in Appendix A).

The numerical counts and content coding on the syllabi will then be entered into the Syllabus

Assessment Instrument.

For step 5, if a technical skill code is present on the syllabus for any assignment or

activity, that syllabus will be given a score of 1 for that specific technical skill. If the skill code is

not present on the syllabus, the syllabus will receive a score of 0 for that technical skill.

Appendix B: Syllabus Assessment Instrument is a detailed example of the research

proposal coding scheme for individual syllabi.

Analysis and discussion points

Once all syllabi have been coded and entered into the Syllabus Assessment Instrument

and SPSS, the initial data analysis will focus on the implicit or explicit use of technical skills in

course objectives and course description along with the technology skills requirements for course

assignment. These data points will make up a bulk of the analysis and the top layer of discovery.

Individually coded syllabi will be aggregated by discipline category (humanities or science) and

date cluster.

13

Please see Appendix C: High-Level Technology Skills Data Analysis for an example of

these aggregated numbers.

The data will be analyzed using SPSS to look for trends, outliers, and relevance in:

high and low levels of general technology skills use

high and low levels of specific technology skills use

explicit and implicit technology requirements

comparison of humanities between date clusters

comparison of sciences between date clusters

comparison of humanities and sciences across date clusters

I will also look for correlations between explicit/implicit and general and specific technology

skills exposure and:

high levels of book or journal usage

print versus digital

syllabus demographics

instructor details.

Conclusion

It is said that imitation is the sincerest form of flattery. There is no doubt that increased

technical literacy development and exposure to computer skills benefits students, faculty, and

their institutions. Based on the debates in the literature related to information gathering,

methodological considerations, and computer technology skills in the humanities, it is assumed

that the successes of the sciences should be imitated by the humanities to realize a new digital

culture on those disciplines.

My proposed research does not seek to understand the reasons for differences in the

comparison (e.g. theories about why instructors in the humanities did or did not use technology).

Instead, I hope to identify and provide additional evidence of these differences and examine

14

whether they are consistent with previous theories about humanist engagement with technology

based on research practice and culture. This study could contribute to our understanding of

pedagogical and cultural differences in the humanities and the sciences that permeate student and

faculty perceptions of the role of technology in their disciplines.

As noted above, I hypothesize that there will be significant differences between the

humanities and the sciences and that the humanities will show less change and less growth

between date clusters. It is my hope that this research identifies some interesting differences

between exposure and use of specific computer technology skills in syllabi in the humanities and

the sciences. Additionally, an analysis of correlating variables and specific technology uses could

also provide a new form of curriculum assessment designed to improve computer technology

exposure in humanities courses.

If there is no significant difference in computer technology skills exposure and use in

humanities syllabi compared to sciences syllabi that is also an interesting result. Results in that

vein would be contradictory to much of the literature related to the use of technology in

academic disciplines. This would indicate the need for additional analysis of exposure to

technology in undergraduate versus graduate level courses and how that translates to

instructional/curriculum focus in new professionals.

If the humanities and sciences are exposing students to computer technology skills in

courses at the same levels, a new theory would need to be developed about the cultural

differences in the humanities affecting current technology usage. This, too, could lead to a new

approach to curriculum assessment.

15

References

Allen, M. (2004). Assessing academic programs in higher education. Bolton, MA: Anker

Publications.

Anderson, S., Blanke, T., & Dunn, S. (2010). Methodological commons: Arts and humanities e-

science fundamentals. Philosophical Transactions of the Royal Society A: Mathematical,

Physical and Engineering Sciences, 368(1925), 3779-3796. doi:10.1098/rsta.2010.0156

Blanke, T., Hedges, M., & Dunn, S. (2009). Arts and humanities e-science—Current practices

and future challenges. Future Generation Computer Systems, 25(4), 480.

doi:10.1016/j.future.2008.10.004

Brown, K. G., Rynes, S. L., Charlier, S. D., & Hosmanek, A. (2013). What do we teach in

organizational behavior? an analysis of MBA syllabi. Journal of Management Education,

37(4), 471. doi:10.1177/1052562913486469

Dahl, C. (2013). PDA and the humanities: Assessing the fit through an examination of the

literature on humanists and e-resources. The Electronic Library, 31(6), 745–752.

Eberly, M. B., Newton, S. E., & Wiggins, R. A. (2001). The syllabus as a tool for student-

centered learning. The Journal of General Education, 50(1), 56–74. Retrieved from

http://jan.ucc.nau.edu/~coesyl-p/syllabus_cline_article_4.pdf

Fink, S. J. B. (2011). An exploratory study on the purpose, structure, format and use of syllabi at

a midwest four-year undergraduate private university (doctoral dissertation). Retrieved

16

from

http://escholarshare.drake.edu/bitstream/handle/2092/1623/dd2011SJBF.pdf?sequence=1

Genoni, P., Merrick, H., & Michele Willson. (2009). E-research and scholarly community in the

humanities. In Nicholas W. Jankowski (Ed.), E-research: Transformation in scholarly

practice (pp. 91-108). New York: Routledge.

Harley, D., & et al. (2006). Use and users of digital resources: A focus on undergraduate

education in the humanities and social sciences. University of California, Berkeley:

University of California. Retrieved from http://escholarship.org/uc/item/8c43w24h

Head, A. J. (2008). Information literacy from the trenches: How do humanities and social science

majors conduct academic research? College & Research Libraries, 69(5), 446.

doi:10.5860/crl.69.5.427

Joe, J. N., Harmes, J. C., & Barry, C. L. (2008). Arts and humanities general education

assessment: A qualitative approach to developing program objectives. JGE: The Journal of

General Education, 57(3), 151. Retrieved from

http://muse.jhu.edu/journals/jge/summary/v057/57.3.joe.html

Kircz, J. G. (2004). Report: E-based humanities and E-humanities on a SURF platform.

Amsterdam: Kircz Research Amsterdam. Retrieved from http://dare.uva.nl/record/1/419739

Krippendorff, K. (1989). Content analysis. International encyclopedia of communication (pp.

403-407). Retrieved from http://repository.upenn.edu/asc_papers/226

17

Madson, Michael B; Melchert, Timothy P; Whipp, Joan L. (2004). Assessing student exposure to

and use of computer technologies through an examination of course syllabi. Assessment &

Evaluation in Higher Education, 29(5), 549–561. Retrieved from

http://epublications.marquette.edu/cgi/viewcontent.cgi?article=1008&context=edu_fac

Mayo, J. (2012). Technology’s role in constructing meaningful knowledge. Pedagogy and the

Human Sciences, 1(2), 8-21. Retrieved from

http://pedagogyandhumanscience.org/files/Mayo_PHSv2.pdf

McDonald, J., Siddall, G., Mandell, D., & Hughes, S. (2010). Two sides of the same coin:

Student-faculty perspectives of the course syllabus Retrieved from

http://windsor.scholarsportal.info/ojs/leddy/index.php/CELT/article/view/3249

Neuman, W. L. (2011). Social research methods: Qualitative and quantitative approaches (7th

ed.). Boston: Allyn and Bacon.

Parkes, M., & Reading, C. (2013). The competencies required for effective performance in a

university e-learning environment. Australasian Journal of Educational Technology, 29(6),

777-791. Retrieved from

http://ascilite.org.au/ajet/submission/index.php/AJET/article/view/38/774

Puckett, C. (2011). Pre-print: Oh, the humanities: Understanding information behavior to foster

information literacy. Emporia: Emporia State University. Retrieved from

https://esirc.emporia.edu/handle/123456789/45

18

Rimmer, J., Warwick, C., Blandford, A., Gow, J., & Buchanan, G. (2008). An examination of the

physical and the digital qualities of humanities research. Information Processing &

Management, 44(3), 1392. doi:10.1016/j.ipm.2007.09.001

White, M. D., & Marsh, E. E. (2006). Content analysis: A flexible methodology. Library Trends,

55(1), 22-45. doi:10.1353/lib.2006.0053

Wikle, T. A., & Fagin, T. D. (2014). GIS course planning: A comparison of syllabi at US college

and universities. Transactions in GIS, 18(4), 574–585. doi:10.1111/tgis.12048

Wolk, R. M. (2004). Creating an outcomes assessment instrument that incorporates information

technology dimensions. Information Systems Education Journal, 4(1) Retrieved from

http://isedj-org.edsigbh.org/4/1/ISEDJ.4%281%29.Wolk.pdf

Wolk, R. M. (2006). How important is student computing ability? the role of information

technology competence in business school accreditation. Paper presented at the , 23

Retrieved from http://www.isedj.org/6/39/Wolk.txt

Zhang, J. (2014). Scholarship before technology: Re-thinking the relationship between

technology and scholars. Journal of Communication and Education, 1(1), 15. Retrieved

from http://www.hkaect.org/jce/issues/JCE_1%281%29.pdf#page=6

19

Appendix A: Computer Technology Skills and Other Variables

20

Appendix B: Syllabus Assessment Instrument

21

Appendix C: High-Level Technology Skills Data Analysis