basic eskills- foundation or frustration - ctdlc · basic eskills—foundation or frustration: ......

Basic eSkills-Foundation or

Frustration:

Diane J. Goldsmith, PhD. David Nielsen, M.A.

George Rezendes, PhD Catherine Manly, MBA

Connecticut Distance Learning Consortium Funded by the Alfred P. Sloan Foundation

May 2006

CONNECTICUT DISTANCE LEARNING CONSORTIUM WWW.CTDLC.ORG

A Research Study of Entering Community College Students'

Computer Competency

2

Basic eSkills—Foundation or Frustration:

A Research Study of Entering Community College Students'

Computer Competency

Diane J. Goldsmith, PhD. Dean of Planning, Research, and Assessment

David Nielsen, M.A.

Director of Institutional Research, Manchester Community College

George Rezendes, PhD Director of Institutional Research, Three Rivers Community College

Catherine Manly, MBA

Former Director of Distance Learning, Manchester Community College

This research project was conducted by the Connecticut Distance Learning Consortium under the direction of Dr. Diane J. Goldsmith. The project was funded by a Grant from the Alfred P. Sloan Foundation. The pilot study and previous work on basic computer literacy was funded by a grant from the Fund for the Improvement of Post-Secondary Education (FIPSE). The simulation software used to develop the test was provided by Course Technology, a division of Thomson Learning. Associated with this study is a technical report on the Validity and Reliability of the Assessment using Item Response Theory. This study was conducted by Dr. George Rezendes and is available from the CTDLC. For more information please contact: Dr. Diane J. Goldsmith Connecticut Distance Learning Consortium 85 Alumni Drive Newington, CT 06111 860.832.3893 [email protected]

© Connecticut Distance Learning Consortium, 2006.

3

TABLE OF CONTENTS

EXECUTIVE SUMMARY ............................................................................................................ 4

Introduction............................................................................................................................. 4 Results..................................................................................................................................... 5 Time to Complete The Assessment ........................................................................................ 6 Validity and Reliability........................................................................................................... 6 Demographic Analysis............................................................................................................ 6 Recommendations................................................................................................................... 7

INTRODUCTION .......................................................................................................................... 9

METHODOLOGY ....................................................................................................................... 11

The Assessment Tool............................................................................................................ 11 Administering the Assessment.............................................................................................. 11

DATA ANALYSIS....................................................................................................................... 12

Score Distributions................................................................................................................ 13 Time to Complete the Basic Computer Skills Assessment................................................... 15 Demographic Analysis.......................................................................................................... 15 Relationship to other Assessment Test Scores...................................................................... 22 Similarity of Test Takers to Admitted Students ................................................................... 23

VALIDITY AND RELIABILITY................................................................................................ 24

Item Characteristics .............................................................................................................. 24 Student Characteristics.......................................................................................................... 25

DISCUSSION............................................................................................................................... 28

RECOMMENDATIONS.............................................................................................................. 29

4

EXECUTIVE SUMMARY

Introduction In the Connecticut Distance Learning Consortium’s efforts over the past 8 years to support eLearning, we have come to believe that students without basic computer skills struggle with the technology instead of the academic content, drop out of courses, and are then excluded, not only from online learning, but also from technology enhanced, classroom-based learning. We had no idea how many of these students there actually were, whether those without skills could self-identify, or whether an assessment tool would be necessary to determine students’ skill levels. To respond to these concerns, the CTDLC surveyed faculty in eleven partner institutions (two and four year, public and private) and asked both classroom based and online faculty what a student needed to know their first day of class. There was a striking amount of unanimity. Using the data from the survey, a Statement of Minimal Technical Literacy for Incoming College Students was derived. Table 1 shows those skills. Table 1 Basic Computer Skills Basic Web Skills

Type Go to a specified web address Open a software application Click on a link Exit a software application Scroll through a webpage Access a CD-ROM Use the back button Locate a saved file Perform a basic search in a search engine Use copy and paste Identify search results

Download and view a file from a webpage Basic Word Processing Skills Print a webpage

Use the enter/return key to create a blank line Tab text to indent a line Basic Email Skills Apply basic formatting: bold, italics, and centering Send an email message Save a file (name the file and locate where to save the

file) Attach a file to an email message

Print a file Receive an email message We then developed two methods of assessing these skills: a self-assessment—we listed the skills and asked students to rate themselves on a scale from “can do easily” to “can’t do”—and two different performance-based assessments—one home designed, and one using Course Technology simulations. We ran two small pilots with students in nine institutions in the winter and spring of 2004. Our preliminary findings indicated that students were not able to self-assess their skills accurately. Many students indicated they could “do easily” things they were unable to do in a performance-based test. More surprising was the discovery that on the performance-based assessments, 10% of the students had few of the basic skills and another 31% had significant weaknesses. Given these preliminary findings, we designed a two-fold project: 1) confirming that a technology literacy assessment is truly needed and 2) creating a valid and reliable assessment tool to measure incoming students’ ability to meet these minimum requirements.

5

With the financial support of the Alfred P. Sloan Foundation and using the simulation software built by Course Technology, we built a 17 item performance-based assessment of the basic skills identified in the Minimal Statement of Technology Literacy. Table 2 shows the tasks included in the Basic Computer Skills Assessment (BCS). We administered it to 2090 students, in five Connecticut Community Colleges, of which 1459 registered for classes in the fall of 2006. Table 2

Skill Set Activity Skill Set Activity Open a window by double-clicking Enter a URL in the Address bar Point to an item Go to a Web page by using links Open a file from within a program Us the scroll bars

Basic Windows Skills

Close a program Find a previously displayed Web page using the back button

Insert text

Basic Internet

Skills

Search the Web using a search engine

Apply character formats Open a new e-mail message window Copy and paste text Reply to an e-mail message Save a document Send a message with an attachment

Basic Word Processing Skills

Print documents

Basic e-mail skills

Results Figure 1 indicates that the majority of students earned a “pass” on the assessment test. While seventy-two percent (72%) earned scores of 82 or higher (14-17 correct responses), it is important to note that only 18% of the students earned a perfect score. Another 22% were deemed to “need review,” with scores between 58 and 76 (10-13 correct). Only 6% of the students are “not meeting basic skills” by receiving a score of 52 or lower (9 or fewer correct).

Figure 1

Basic Computer Skills AssessmentSummer 2005

Score Distribution

0

50

100

150

200

250

300

0 5 11 17 23 29 35 41 47 52 58 64 70 76 82 88 94 100

Assessment Test Score

Num

ber o

f Stu

dent

s

Not meeting basic skills

Pass

Need Review

6

Time to Complete The Assessment While we gave students 20 minutes to complete the assessment, students who passed took an average of 7 minutes. Students who scored in the “need review” to “not meeting basic skills” range took an average of 10 minutes. Thus, the assessment provided a relatively short means of obtaining a snapshot of students’ skills.

Validity and Reliability Using Item Response Theory, we were able to discover which items required the most ability. These are arranged from most difficult to least:

Send an email message with an attachment (most difficulty) Reply to an email message Enter a URL in the address bar Copy and paste text Use a search engine to find specific information on the web.

We were also able to discern items which could be eliminated because they measured the same ability levels. Thus the assessment could be shortened, or additional items could be added to increase the difficulty of the assessment.

Demographic Analysis Scores from the Basic Computer Skills Assessment were analyzed with the available student demographic information. That analysis indicates that:

Men are significantly more likely to earn a passing score than women. Younger students (19 and younger) are more likely to earn a passing score than students

over 25. Students with higher family incomes are more likely to earn a passing score than

students with lower family incomes. Students who attended a high school where fewer students are eligible for free/or

reduced price meals are more likely to earn a passing score than those attending schools where more students receive such meals.

There does not appear to be a correlation between prior college experience and scores on this assessment. However, students who complete tech prep credits scored higher than students with transfer credits.

While there are different pass rates for different ethnic groups, when ethnicity and income are entered into a regression analysis, only income plays a role in predicting scores.

Students who score higher in the reading and writing Accuplacer assessments also score higher on the Basic Computer Skills Assessment. There is no correlation between math scores and scores on the BCS.

Students who registered for classes for the Fall of 2005 have higher scores on the BCS than those who did not register.

7

Recommendations 1. The basic computer skills test provided by Course Technology provides a short, reliable, and

valid test of the skills identified by faculty as necessary for students on their first day of class. We recommend that institutions require such an assessment for all of their entering students.

2. The items chosen for the test were based on a survey done of faculty in the spring of 2003.

Faculty were asked to identify the computer skills students needed to know when they entered college. Given the speed at which technology is increasingly made a part of higher education, it is recommended that such a survey be repeated in the near future to ensure that the tasks measured by this assessment are still those faculty believe are essential.

3. The Item Response Theory analysis indicates that several pairs of the tasks (item 1 and item

6) provide redundancy in the assessment and thus one of the tasks may be eliminated to either reduce the time of the test or to add additional items of greater difficulty. Such modifications could be considered after an analysis of the survey in recommendation 2.

4. While students were allowed up to twenty minutes to complete the test, the fact that students

in the bottom two groups averaged ten minutes, indicates that a shorter test completion time could be used.

5. The CTDLC has created a free, self-paced, web-based, four-module Basic Online Computer

Skills program (http://www.ctdlc.org/remediation/ ). Institutions should strongly recommend such a course to those students who fall into the “need review” category.

6. As only 18% of the students – less than 1/5 – were able to answer all of the questions, even

those students who pass should be advised to review those basic skills where the test indicates they need remediation. The student can then be directed to those particular modules in the Basic Online Computer Skills course.

7. The Item Response Theory analysis indicated that even proficient students may have

difficulty with two tasks—replying to an email message and attaching a document to an email-- even if they have mastered the others. Thus faculty can expect that a number of incoming students will have initial difficulty sending them papers electronically. If this will be required in the cures, faculty may wish to put particular emphasis on the computer assistance that is available at the college.

8. Institutions should consider creating an Introduction to Basic Computer Skills Orientation

course specifically designed for the approximately 6% of those students whose assessment scores indicate they do not have the basic skills identified as necessary for a successful entering student.

9. In this research study, older students score lower than younger students. Care should be

taken in designing orientation courses to remediate these basic skills, as noted in recommendation five, to schedule them at times that are convenient for adult learners.

8

10. Because the research shows a strong positive correlation between the computer skills assessment scores and the English and Reading Accuplacer scores, consideration should be given to adding appropriate technology instruction and requirements to the curriculum of developmental English courses.

11. The fact that there exists a correlation between basic computer skill scores and the high

school a student attends suggests that colleges might work more closely with high schools showing lower student scores to ensure that basic online computer skills are incorporated into their curriculum.

9

BASIC eSKILLS: FOUNDATION OR FRUSTRATION

INTRODUCTION In the Connecticut Distance Learning Consortium’s efforts over the past 8 years to support eLearning, we have come to believe that students without basic computer skills struggle with the technology instead of the academic content, drop out of courses, and are then excluded, not only from online learning, but also from technology enhanced, classroom-based learning. While there is much discussion regarding “digital natives,” younger students with highly developed technical skills entering higher education, we believed that many of the non-traditional-aged students, and many students from poor or working class families who enter community colleges and who take online courses from both two and four year institutions, were entering colleges without the most basic technology skills. We had no idea how many of these students there actually were, whether those without skills could self-identify, or whether an assessment tool would be necessary to determine students’ skill levels. To respond to those concerns as part of a project funded by the Fund for the Improvement of Post Secondary Education (FIPSE), in 2003 the CTDLC surveyed faculty in eleven partner institutions (two and four year, public and private) and asked both classroom based and online faculty what a student needed to know their first day of class. There was a striking amount of unanimity. Faculty identified basic technology skills in three areas: word processing, email, and internet browsing. Using the data from the survey, a Statement of Minimal Technical Literacy for Incoming College Students was derived. Table 1 shows those skills. Table 1 Basic Computer Skills Basic Web Skills

Type Go to a specified web address Open a software application Click on a link Exit a software application Scroll through a webpage Access a CD-ROM Use the back button Locate a saved file Perform a basic search in a search engine Use copy and paste Identify search results

Download and view a file from a webpage Basic Word Processing Skills Print a webpage

Use the enter/return key to create a blank line Tab text to indent a line Basic Email Skills Apply basic formatting: bold, italics, and centering Send an email message Save a file (name the file and locate where to save the

file) Attach a file to an email message

Print a file Receive an email message We then developed two methods of assessing these skills: a self-assessment—we listed the skills and asked students to rate themselves on a scale from “can do easily” to “can’t do”—and two

10

different performance-based assessments—one home designed, and one using Course Technology simulations. We ran two small pilots with students in ten institutions in the winter and spring of 2004. Eight of the institutions gave the students the CTDLC self-assessment and the CTDLC-created performance-based test test. Our preliminary findings indicated that students were not able to self-assess their skills accurately. Many students indicated they could “do easily” things they were unable to do in a performance- based test. Two institutions gave the Course Technology Test. Using that performance-based test we found that 10% of the students had few of the basic skills and another 31% had significant weaknesses. Given these preliminary findings, which indicated that students were not able to self-identify their skills, we were interested a two-fold project: 1) confirming that a technology literacy assessment is truly needed and 2) creating a valid and reliable assessment tool to measure incoming students ability to meet these minimum requirements. With the financial support of the Alfred P. Sloan Foundation and using the simulation software built by Course Technology, we were able to create an assessment tool, administer it to entering students in five Connecticut Community Colleges, and use the results to see if there were any significant correlations between basic computer skills, student demographics, and student scores on math, English, and reading placement tests. We also analyzed the assessment instrument itself to ensure that it is psychometrically sound--reliable, valid and an accurate measure of ability--using Item Response Theory (IRT). IRT is a statistical modeling theory which allows us to understand student abilities and the characteristics of the test items as independent from each other. This allowed us to ensure that the test adequately measures a range of abilities from low to high. It also allowed us to recommend the elimination of questions which measure the same abilities, thus creating a shorter test which is important to institutions that already require lengthy placement testing for their new students.

11

METHODOLOGY

The Assessment Tool Using the Course Technology simulation software, we built a 17 item performance-based assessment of the basic skills identified in the Minimal Statement of Technology Literacy. These were the same skills tested in the Winter 2004 pilot. Table 2 shows the tasks included in the Basic Computer Skills Assessment (BCS). These tasks are the same as were required in the initial pilot. Because the software completely simulates the Windows, Word, Internet, and email environments, students can respond to these items using any of the tools normally available in these programs, including drop down menus, key strokes, or icons. Table 2

Skill Set Activity Basic Windows Skills Open a window by double-clicking Point to an item Open a file from within a program Close a program Basic Word Processing Skills Insert text Apply character formats Copy and paste text Save a document Print documents Basic Internet Skills Enter a URL in the address bar Go to a web page by using links Use the scroll bars Find a previously displayed web page using the back button Search the web using a search engine Basic e-mail skills Open a new e-mail message window Reply to an e-mail message Send a message with an attachment

Administering the Assessment An invitation was sent to the 12 Connecticut Community Colleges and the baccalaureate granting institutions that are members of the CTDLC offering them the opportunity to participate in this project. A $1000 stipend was offered to defray some of the costs of participation. We were hoping to collect data from 8-10 institutions. The Project Test Coordinator met with the testing coordinators of the Community Colleges, and in the end, five institutions (Manchester CC, Middlesex CC, Naugatuck Valley CC, Three Rivers CC, and Tunxis CC) agreed to add the Basic Computer Skills Assessment (BCS) to their set of Accuplacer assessments in reading, English, and math, required of all students applying to each institution. Students who were scheduled for testing between April 1, 2005 and July 31, 2005 in a college testing center also

12

took the Basic Computer Skills Assessment test. This assessment was given after the Accuplacer assessments. The Course Technology test allows time limits to be enforced, a significantly helpful factor for college testing administrators. During the pilot when we also used a time-unlimited home designed test, we observed that students with very weak computer skills did not do significantly better when they took 40 minutes to work on the test than when they had at 20 minutes. This indicated that a time-intensive test was not needed to gain an accurate assessment of computer skill level, which makes it easier to fit into the college testing sessions. The BCS assessment was limited to 20 minutes. The assessment tool allows the test makers to set the number of tries a student has to complete the test. For this assessment, we allowed a student three attempts before their response was scored as incorrect. A small number of students were observed to miss one question while they figured out the mechanics of going through the test. It could be helpful in the future to revise the test instructions or have a sample question prior to the scored questions so students who are apt to click the mouse a lot do not miss a question while they figure out how to take the test. During the pilot, the questions were presented in a fixed order, but during the Summer 2005 testing the questions were randomly presented in each of the skill areas to minimize the possibility of students copying each other. We know of no instances of cheating on this test in either 2004 or 2005.

DATA ANALYSIS A total of 2090 students took the assessment test. A total of 1459 of those students completing the assessment actually registered for classes at one of the Community Colleges for the Fall 2005 semester. The data from the test was matched to a series of data files from the Community Colleges. Demographic and academic information was available for 1258 of the students. We took the following steps to match the test scores to official CT-CC data files that included demographic, socio-economic, and placement test information:

• Students who attempted the BCS test more than once were identified, and all but one record was eliminated; in most cases, it was clear that one attempt was aborted

o For eight students it was less clear which score to keep; in six cases the higher of the two scores was kept; in two of the cases the score was the same, so the score from the later date was used.

• Banner IDs were manually adjusted to either add the “@” sign, add leading zeroes, eliminate leading zeros, or eliminate extra digits.

• Eighty-four students were manually mapped to a Banner ID using the name and institutions from official CT-CC data files.

• The methods above did not result in a Banner ID match for 201 students; all but a few completed the BCS at Tunxis CC; it is likely these students did not complete an admissions application, and therefore were not entered into the swrxA11 file used in these analyses.

13

Score Distributions To allow for a comparison to the initial pilot, the same three categories of results were used:

Pass: 14-17 items done correctly (82% to 100% correct) Need Review: 10-13 items done correctly (58% to 76% correct) Not Meeting Basic Skills: 9 or fewer items done correctly (0 to 52% correct).

Figure 1 indicates that the majority of students earned a “pass” on the assessment test. While, seventy-two percent (72%) earned scores of 82 or higher it is important to note that only 18% of the students earned a perfect score. Another 22% were deemed to “need review,” with scores between 58 and 76. Only 6% of the students are “not meeting basic skills” by receiving a score of 52 or lower.

Figure 1


Score Distribution

0

50

100

150

200

250

300

0 5 11 17 23 29 35 41 47 52 58 64 70 76 82 88 94 100

Assessment Test Score

Num

ber o

f Stu

dent

s

Not meeting basic skills

Pass

Need Review

There were only minor differences by host institution, with no statistically significant differences.

14

The results of the Summer 2005 test show a marked improvement versus the 2004 pilot test results (Figure 2). The difference in the share of students earning a “passing” score is significantly higher in the 2005 administration.

Figure 2

Some of this difference may be attributable to differences in the samples. The 2005 sample includes a larger share of men and a larger share of young students, both variables that are tied to higher test scores in the 2005 assessment. However, it is also probable that, as computers become cheaper and schools and work require more computer knowledge, skill levels are increasing.

Basic Computer Skills AssessmentSummer 2005 vs. Winter 2004 Pilot

Score Distribution

10% 31%

60%

6%22%

72%

0%20% 40% 60% 80%

100%

Not Meeting Basic Skills52 or lower

Need Review58-76

Pass 82 or higher

Winter 04 Pilot Summer 05 Pilot

15

Time to Complete the Basic Computer Skills Assessment Students were allowed up to twenty minutes to complete the assessment test. As one might expect, higher scores are associated with faster completion times (Figure 3). The strong negative correlation (-0.39) suggests that students with the proficiency to pass the test can work through the assigned tasks at a faster rate than students with lower proficiencies. But students who need review or lack basic skills, on average, completed the test in almost the same time.

Figure 3

Basic Computer Skills AssessmentAverage Elapsed Time to Compete Test by Score Grouping

0:10:14 0:10:10

0:07:02


Need Review58-76

Pass82 or higher

The fact that even students at the bottom averaged approximately ten minutes to take the test indicates that a shorter test completion time could be used without sacrificing accuracy.

Demographic Analysis There are clearly differences in assessment test scores by demographic categories. Men score better than women, younger students score better than older students, and socio-economic differences are evident as well. Students with higher family incomes score better than those with lower family incomes.

16

Men are significantly more likely to earn a passing score than women. Further, the average score for Men (84.6) is significantly higher than the average score among Women (82.5) (figure 4).

Figure 4


Test Score Distribution by Gender70% 75%

0%

20%

40%

60%

80%

Women Men

Shar

e of

Stu

dent

s

Not Meeting Basic Skills52 or low er

Need Review58-76

Pass82 or higher

Younger students, particularly those age 19 and younger, are more likely to receive passing scores than older students. The average test scores by age group, as shown below (figure 5), demonstrate this relationship between age and proficiency.

Figure 5


Average Test Score by Age Group85.1

83.2

78.076.5

7274767880828486

19&younger 20-21 22-24 25+

Ave

rage

Tes

t Sco

re

17

High school graduation year does not show the same strong positive relationship with BCS scores as seen with age (figure 6). Time since high school graduation does not have a strong correlation with English or Math placement scores either. The average BCS, English, and Math test scores are all lower for the class of 2005 vs. graduates from 2003 or 2004.

Figure 6


Average Test Score by High School Graduation Year

79.1

81.1

85.985.3

84.2

74

76

78

80

82

84

86

88

Pre-2000(n=154)

2000-01-02(n=68)

2003(n=52)

2004(n=93)

2005(n=869)

Ave

rage

Tes

t Sco

re

18

There are considerable differences in pass rates by the student’s high school (Table 3). In the broadest of terms, the schools with the higher pass rates tend to be located in suburban areas, while the schools with the lower pass rates tend to be located in Connecticut’s large and medium size cities. Students who earned a GED rather than a traditional high school diploma were significantly less likely to earn a passing score on the BCS test.

Table 3

High-SchoolCount of Students

Average BCS

ScorePass Rate

Grand Total 1459 82.8 72%Norwich Free Academy 29 89.7 83%Naugatuck HS 22 87.2 82%Manchester HS 48 88.4 81%Newington HS 20 84.7 80%South Windsor HS 22 86.4 77%Middletown HS 24 85.5 75%East Hartford HS 55 82.7 75%Rockville HS 25 82.1 72%Platt HS (Meriden) 23 78.4 70%Maloney HS (Meriden) 22 79.1 68%Bristol Central HS 20 82.0 65%Bulkeley HS (Hartford) 37 74.3 49%

Out Of State 50 82.2 70%Gen. Educ. Dev. (GED Equiv.) 61 74.1 48%Out Of Country 23 57.7 30%

19

In order to look more closely at these different pass rates by High School, we used data from the from Connecticut Strategic School Profiles for each of the “major” feeder High Schools. As a surrogate for family income we used the share of students at each high school that are eligible for free or reduced-price meals. There is a strong negative correlation with the share of students from that high school that earned a “pass” on the BCS Assessment and the percentage that are eligible for free or reduced meals (figure 7).

Figure 7

HIGH SCHOOL ANALYSISBCS Pass Rate vs.

HS's Share Eligible for Free/Reduced-Price Meals

0102030405060708090

Naugatu

ck H

S

Norwich

Free A

cadem

y

Manch

ester H

S

Newington

HS

South

Windso

r HS

Middlet

own H

S

East H

artfor

d HS

Rockvill

e HS

Platt H

S Meri

den

Malone

y HS M

eriden

Bristol

Cen

tral H

S

Bulkeley H

S Hart

ford

% Eligible for Free/Reduced-Price Meals BCS Pass Rate

There does not appear to be any link between computer proficiency and prior college experience (figure 8). Students who complete Tech Prep credits and continue on to community colleges score higher than other students with transfer credits. This difference, however, is not significant.

Figure 8


Average Test Score by Admit Type

83.4 83.3 86.0

50

60

70

80

90

New Admit 1st TimeCollege Student

Transfer from AnotherCollege

Tech PrepPostsecondary Student

Ave

rage

Tes

t Sco

re

20

There are statistically significant differences in pass rates by ethnic group (figure 9). White students are far more likely to earn passing scores than Black or Hispanic students. However, there is a strong interaction effect with income, as detailed below (figure 10).

Figure 9


Test Score Distribution by Ethnicity

13% 11%3%

51% 57%

79%

0%20%

40%60%

80%100%

Black, non-Hispanic Hispanic White, non-Hispanic

Shar

e of

Stu

dent

s


Need Review58-76

Pass82 or higher

There appears to be a strong positive relationship between family income and basic computer skills proficiency (figure 10). The average incomes by score group, as shown below, indicate this, as does the strong positive correlation (+0.22) between score and family income.

Figure 10


Average Family Income by Test Score Group

$27,322

$42,178

$51,520

$-

$10,000

$20,000

$30,000

$40,000

$50,000

$60,000


Need Review58-76

Pass82 or higher

Ave

rage

Fam

ily In

com

e

When ethnicity and family income are both used in a regression analysis, ethnicity did not play a role in any of the regression models. This suggests that socio-economic status is a better predictor of basic computer skills than race

21

There is clearly a strong interaction effect between ethnicity and income. Within each of the ethnic groups the relationship between computer proficiency and average family income is apparent (figure 11).

Figure 11


Average Family Income by Ethnic Group and Test Scores

$21,077 $17,682

$37,844$38,496 $40,110

$55,896

$-

$10,000

$20,000

$30,000

$40,000

$50,000

$60,000

Black, non-Hispanic Hispanic White, non-HispanicAve

rage

Fam

ily In

com

e


Need Review58-76

Pass82 or higher

There also appears to be an interaction effect between age and ethnicity that may have an impact on the BCS results. White and Hispanic students are significantly more likely to fall into the 19 & younger age group than the Black students in the sample. As seen earlier, younger students outperform older students in this test of basic computer skills (figure 12).

Figure 12


Interaction of Age and Ethnicity in Sample

64%

11%5%

20%

72%

8% 3%17%

74%

9% 5%12%

0%

20%

40%

60%

80%

Age 19 & Younger Age 20-21 Age 22-24 Age 25 & Older

Shar

e of

Stu

dent

s

Black, non-Hispanic Hispanic White, non-Hispanic

There do not appear to be interaction effects between age and gender, or gender and ethnicity.

22

Relationship to other Assessment Test Scores There is a strong positive correlation between Basic Computer Skills Assessment test scores and the students’ placement test scores in Math, English and Reading (figure 13). The strongest correlation is with the English test (+0.42), generally considered a “sentence skills” assessment. Reading placement test scores have the next strongest correlation (+0.38). Math placement test scores have the weakest correlation of the three, but still suggest a strong positive relationship (+0.25).

Figure 13


Correlation between BSA Score and Placement Test Scores

0.252

0.423 0.382

0.00

0.10

0.20

0.30

0.40

0.50

BCS Score vs.Math

BCS Score vs.English

BCS Score vs. Reading

Corr

elat

ion

Val

ue

The chart (figure 14) below maps specific scores on the Basic Computer Skills Assessment test to the average placement test scores for those students. While the relationship appears to be less clear amongst those who “do not meet basic skills” on the BCS test (score 52 or lower), this is likely due to smaller base sizes at several of these lower BCS test score data points.

Figure 14


Average Math, English and Reading Placement Test Scoresvs. BCS Test Scores

0

20

40

60

80

100

0 17 29 41 52 64 76 88 100

Basic Computer Skills Assessment Score

Ave

rage

Pla

cem

ent

Test

Sco

re

EnglishReadingMath

23

Similarity of Test Takers to Admitted Students Not all students who were applying for admission took the BCS. Students who applied at the last minute and were tested in August were excluded from this group for logistical reasons. However, when students who took the test were compared with the pool of applicants who are new first-time college students, they were found to be quite similar, though there were some notable differences. (Table 4) The BCS sample includes fewer students of color and a higher share of young students. Because older students tended to have few basic computer skills, the BCS data may slightly over represent the percentage of students who enter college with these basic skills.

Table 4 All Applicants BCS Sample

New 1st time college students 53% 77%

TOTAL 13101 1459

All NEW Applicants BCS SampleAverage Math Placement Score 47.3 49.2Average English Placement Score 79.2 80.5Average Reading Placement Score 71.5 72.7

% Female 53% 55%

% White 68% 75%% Black 11% 9%% Hispanic 16% 13%

19&younger 67% 73%20-21 10% 9%22-24 7% 5%25+ 17% 14%

TOTAL 6,880 1,459

24

VALIDITY AND RELIABILITY Test Validity and Reliability were analyzed using Item Response Theory1 to estimate the item characteristics of the assessment as well as basic computer literacy ability characteristics of the students. The characteristics of both the assessment and students were then examined relative to student demographic information. Item Response Theory (IRT) is an alternative testing theory that models individual responses at the item level thus overcoming several of the identified shortcomings of the more traditional testing theory known as Classical True Score Theory (CTST). Specifically, IRT provides item characteristics that are not group-dependent, scores reflect individual ability that are not test-dependent, an assessment of score reliability which is not dependent on parallel tests, and a measure of precision for each ability score. A significant advantage of IRT is the independence of item characteristics and student characteristics. In practical terms this allows the researcher to separate the difficulty of the assessment from the ability of the students.

Item Characteristics Table 5 provides a list of basic computer skills (BSC) tasks ordered by item difficulty from easiest to most difficult. The five most difficult tasks in this assessment are associated with email and internet tasks, with the exception of copying and pasting text. They include:

Send an email message with an attachment (most difficulty) Reply to an email message Enter a URL in the Address bar Copy and Paste Text Use a search engine to find specific information on the web.

Figure 15 provides the item characteristic curve (ICC) for each of the items contained on the BSC assessment. The Item Characteristic Curve (ICC) for each item provides the probability that an individual examinee with an ability level of θ will get the item correct. Examining all ICC’s for a particular assessment on a single ability continuum provides information about the validity of the assessment in that items should adequately span the ability continuum to satisfactorily distinguish between the ability levels of the respondents. The curves of items 13 (reply to an email message) and item 14 (attach a document to an email) indicate that even proficient students may have difficulty with these two tasks even if they have mastered the others. Thus faculty can expect that a number of incoming students will have initial difficulty sending them papers electronically. If this will be required in the courses, faculty may wish to put particular emphasis on the computer assistance that is available at the college.

1 Due to the technical nature of Item Response Theory, we have excerpted a portion of the methodology and findings from the full report. The full report including the estimation methodology is available from the Ct Distance Learning Consortium. See the information on CTDLC in the front of this report.

25

The ICC’s will also provide an indication of items that are providing redundant information in that they occupy similar locations on the ability continuum. An example of two items that appear to be providing redundant information for the BSC assessment are items 1 and 6 which have difficulties of -1.79 and -1.78 and discrimination values of 0.99 and 0.82, respectively. While these two tasks, “Use the default settings to print the document” (item 1) and “Apply Bold formatting” (item 6), appear to be measuring different skills, they are in fact measuring the same ability level. They therefore provide redundancy in the assessment and thus one of the tasks may be eliminated to either reduce the time required to complete the assessment or to add an additional task with increased difficulty. An examination of Table 4 shows several sets of items falling in very close proximity relative to difficulty that could be re-examined with an eye for making modifications to the existing BSC assessment to both reduce assessment time and quite possibly add additional difficulty to the assessment.

Student Characteristics Further analysis of the 1047 respondents contained in the selected sample showed that 535 (51.1%) respondents did not register for classes at any of the community colleges during the fall 2005 term while 512 (48.9%) of the selected respondents did. A comparison of the distributions between respondents that registered for classes during fall 2005 and those who did not showed a significant difference. Those students registering for classes during the fall 2005 semester contained more students in the highest ability group and fewer students in the lowest ability group.

Table 5 Basic computer literacy assessment tasks ordered by item difficulty

Difficulty Task Number Skill set Activity Task

-2.36 4 Getting Started with Windows

Open a window by double-clicking Open the My Computer window by double-clicking. [578]

-2.26 5 Getting Started with Windows Point to an item Point to the My Computer icon on the desktop. [571]

-2.26 17 Working with Programs Close a program Exit the WordPad program. [576]

-1.93 3 Getting Started on the Internet Scroll bars Scroll down to read the description of James Brown's Greatest Hits. [973]

-1.91 16 Working with Files Open a file from within a program From within WordPad, open the Sample Text.rtf file. [589]

-1.79 1 Formatting Documents Print documents Use the default settings to print the document. (Note: The document will not actually print.) [167]

-1.78 6 Inserting and Modifying Text Applying character formats Apply Bold formatting to the text "50%" in paragraph 1. [244]

-1.75 2 Getting Started on the Internet Go to a Web page by using links Navigate to the Contact Us page of the Downtown Records Web site.

-1.64 9 Managing Documents Save a document Save the current document with the same filename to the "SAM XP" folder. [172]

-1.63 15 Using the Elements of a Web Browser

Finding a previously displayed Web page

Return to the Downtown Records main page by going one page back in the browser. [994]

-1.51 8 Inserting and Modifying Text Insert text Insert the word "Northwest" one space after the word "Pacific" in the title text.

[101]

-1.44 12 Using e-mail Open a new e-mail message window Open a new Untitled Message - Microsoft Word window. [602]

-1.25 11 Searching the Internet Searching the Web using another search engine

Search for more sites that sell records using Google (www.google.com). Use 'records' as your search term. [1031]

-1.06 7 Inserting and Modifying Text Copy and paste text Copy the line containing the text "Paintings and Drawings". Paste it on the line

below "Heinrich Strubel". [103]

-0.37 10 Navigating Using the Address bar

Entering a URL in the Address bar

Navigate to the Downtown Records main page by entering the URL in the Address bar. Use the URL www.course.com/downtown_records. [1009]

0.64 13 Using e-mail Reply to an e-mail message Create a reply with the text "I completely support this new product line." as a reply to New Products message. Send the reply. [604]

0.76 14 Using e-mail Send a message with an attachment

Send the Last Year's Product Line document (from the My Documents folder) as an attachment to this e-mail message. [603]

Figure 2. Item Characteristic Curves for Basic Computer Skills Assessment

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

-3.000 -2.000 -1.000 0.000 1.000 2.000 3.000

Ability

Prob

abili

ty o

f a c

orre

ct re

spon

se

Item#1 Item#2 Item#4 Item#5 Item#6 Item#7 Item#8 Item#9 Item#10Item#11 Item#12 Item#13 Item#14 Item#15 Item#16 Item#17

Figure 15:

28

DISCUSSION It is clear that most students do bring a foundation of basic computer skills to their first day of class, but it is not necessarily a strong foundation. Fewer than one-fifth of the students tested were able to complete all of the tasks correctly. And these are the very basic tasks of opening programs, using basic word processing skills, searching the internet, and sending an email with an attachment, all skills increasingly necessary for college success. For the bottom 6%, those who lack even these basic skills, the computer requirements of classes which use email, learning management systems, and require papers that are word processed, create one more frustration as they begin their college experience. There are two caveats to these results. Language may present some barriers. Students may know how to do certain tasks but may not be familiar with the language used in the assessment. The testing coordinators reported that ESL students had problems with the test, again possibly more related to their English skills than to their computer skills. It also appears that the numbers of students who do not have these skills or need remediation have decreased in the year and a half between the pilot assessment and this assessment. The demographics of the two groups of students may not be exactly the same because the two assessments were given at different times of the year and in different colleges, but given the rapid integration of technology into both K-12 and the workplace, this increase in skills seems likely. Some may argue then, that if we wait long enough, this issue will disappear; and it may. But the demographics indicate that it will disappear faster for younger students, students who have higher family incomes, students who don’t have GEDs, and students who show proficiency in reading and writing. For those older, less proficient, students from poorer families, the lack of these skills will constitute one more barrier to their success in college. Identifying and remediating these requisite skills before students enter their first college classroom would remove one of these barriers. The simulation software offered by Course Technology provides a reliable and valid method of assessing students’ abilities to perform these tasks. The pilot study indicated that students were unable to self-identify their computer abilities, so providing students with an assessment that accurately simulates the internet, word processing, windows, and email environment is invaluable. The simulation allows students to use any form of response (keystrokes, right clicking on the mouse, menus) to respond to the prompt and can be set so that students have a specific number of tries before they are marked as unable to do the task. While we created a very specific, time limited set of tasks for this assessment, Course Technology software would allow colleges to customize assessments to include more difficult tasks to be used for placement rather than just assessment. One problem with this software, however, is that its shear robustness requires a significant download and, as a result, is most useful for testing completed in a testing center. The software can be downloaded into a home computer, but we assumed that if a student had the skills to accomplish the download, they would not need the test. Unless the company creates some sort of web-based testing, this

29

assessment would not be appropriate for students who take courses online and/or are too far from the college to travel to a testing center.

RECOMMENDATIONS 1. The basic computer skills test provided by Course Technology provides a short, reliable, and

valid test of the skills identified by faculty as necessary for students on their first day of class. We recommend that institutions require such an assessment for all of their entering students.

2. The items chosen for the test were based on a survey done of faculty in the spring of 2003.

Faculty were asked to identify the computer skills students needed to know when they entered college. Given the speed at which technology is increasingly made a part of higher education, it is recommended that such a survey be repeated in the near future to ensure that the tasks measured by this assessment are still those faculty believe are essential.

3. The Item Response Theory analysis indicates that several pairs of the tasks (item 1 and item

6) provide redundancy in the assessment and thus one of the tasks may be eliminated to either reduce the time of the test or to add additional items of greater difficulty. Such modifications could be considered after an analysis of the survey in recommendation 2.

4. While students were allowed up to twenty minutes to complete the test, the fact that students

in the bottom two groups averaged ten minutes, indicates that a shorter test completion time could be used.

5. The CTDLC has created a free, self-paced, web-based, four-module Basic Online Computer

Skills program (http://www.ctdlc.org/remediation/ ). Institutions should strongly recommend such a course to those students who fall into the “need review” category.

6. As only 18% of the students – less than 1/5 -- were able to answer all of the questions, even

those students who pass should be advised to review those basic skills where the test indicates they need remediation. The student can then be directed to those particular modules in the Basic Online Computer Skills course.

7. The Item Response Theory analysis indicated that even proficient students may have

difficulty with two tasks—replying to an email message and attaching a document to an email-- even if they have mastered the others. Thus faculty can expect that a number of incoming students will have initial difficulty sending them papers electronically. If this will be required in the courses, faculty may wish to put particular emphasis on the computer assistance that is available at the college.

8. Institutions should consider creating an Introduction to Basic Computer Skills Orientation

course specifically designed for the approximately 6% of those students whose assessment scores indicate they do not have the basic skills identified as necessary for a successful entering student.

30

9. In this research study, older students score lower than younger students. Care should be

taken in designing orientation courses to remediate these basic skills, as noted in recommendation 5, to schedule them at times that are convenient for adult learners.

10. Because the research shows a strong positive correlation between the computer skills

assessment scores and the English and Reading Accuplacer scores, consideration should be given to adding appropriate technology instruction and requirements to the curriculum of developmental English courses.

11. The fact that there exists a correlation between basic computer skill scores and the high

school a student attends suggests that colleges might work more closely with high schools showing lower student scores to ensure that basic online computer skills are incorporated into their curriculum.