cognitive styles and online behavior of novice searchers

Informorion Processing & Managemenf Vol. 26, No. 4. pp. 503-510. 1990 0306-4573/90 $3.00 + .Ml Printed in Great Britain. Copyright 0 1990 Pergamon Press plc

COGNITIVE STYLES AND ONLINE BEHAVIOR OF NOVICE SEARCHERS*

ELISABETH LOGAN School of Library and Information Studies, Florida State University,

Tallahassee, FL 32306, U.S.A.

(Received 17 February 1989; accepted in final form 23 October 1989)

Abstract-Studies of online searching have acknowledged the existence of a wide variety in online search process and outcome variables among individual searchers. Iden- tifying variables associated with these differences could influence the hiring and training of online searchers and impact the design of heuristics for expert systems for information retrieval. Novice searchers at Florida State University were studied to determine relationships between cognitive styles and five measures of online behavior. Results indicate a consistent relationship between placement in quadrants of the Learning Style Inven- tory and high and low mean group scores. Compared with online studies by Saracevic and Woelfl, however, the results from the FSU study are only partially confirmed.

1. INTRODUCTION

One of the most puzzling findings that appears repeatedly in studies of online behavior is the lack of similarity demonstrated by searchers in both process and outcome of online searches. This seems to occur despite comparable levels of searcher training and experience and often occurs when searching in response to the same query. Despite a plethora of investigations, there has been a notable lack of definitive results identifying the characteristics associated with these differences. At the same time, Williams [l], Saracevic [2], and others [3-71, acknowledge the searcher interface as a critical and largely poorly understood factor in information retrieval.

Opinions vary about the feasibility of adapting automated systems to the search process. According to Swanson [8], “It is not possible to instruct a machine to translate a request into an adequate set of search terms. Search terms are hypotheses, inventions, or conjectures; there are no rules.” Whereas Croft and Thompson [9] in describing their 13R system for document retrieval say, “It supports inference based on statistical techniques and domain knowledge . . . the types of activities undertaken by the system during a search session are similar to those done by a human intermediary.”

Nevertheless, it does appear that if we are to provide a viable basis for the variety of automated services and products being devised to assist the system user, identifying and formalizing the heretofore largely intuitive online search processes becomes a priority. Front ends, interfaces, menus, intermediary systems, gateways, whether devised as expert systems or not, rely heavily on an understanding of the user/system interface. The rela- tive success of these user-friendly devices lies not only with increased technologic capability but with the degree to which the user’s potential interaction with the system is understood. Insight into the ways humans approach the online searching environment as well as the possibility of determining which, if any, of the contributing variables can be identified prior to the active online experience could be critical in the design of such systems.

2. BACKGROUND STUDIES

Among studies that find differences in online behavior between searchers searching the same question are those by Woelfl [lo] and Saracevic and Kantor [I 11. Woelfl’s study of experienced MEDLINE searchers reports a surprising lack of similarity in process and

*Requests for reprints of this article should be addressed to the author at the above address.

503

504 E. LOGAN

outcome among experts searching the same question. The recent study by Saracevic and Kantor investigating a variety of factors associated with query analysis as well as online process and outcome measures reports less than 25% overlap in term selection for 56.4% of searchers, and even less overlap between items retrieved-20% for 75.3070 of searchers.

In an attempt to explain diversity among online searches, studies have tried to identify individual characteristics that can be associated with facets of online behavior. Fenichel’s 1981 study indicates very little substantive difference in online behavior between novice and expert searchers. Experts had higher recall scores and their unit cost was lower, but novices performed as well as experts in terms of precision [12]. Bellardo’s investigation of students in online courses at six library schools evaluates their search performance in light of measures of creativity, intelligence, and personality, but finds very little correlation among the measures [13].

Fidel has studied the searching processes of five searchers in detail and notes the differences between “operationalists” and “conceptualists” and the way in which their searches are structured 1141. Brindle examines correlations between cognitive style, familiarity with the system and the subject area and patterns of user behavior as described by Penniman. Here again no significant differences are identified [ 151. Borgman in a pilot study looking at the effect of individual differences in online catalog use, finds that of a number of factors evaluated, college major appears to be the one related to variables associated with aptitude for information retrieval and programming [ 161. Currently at Syracuse, Nilan [17] and others are engaged in studies of the user processes of decision-making and the crite- ria used to evaluate information decisions.

3. CURRENT STUDY

In the study by Woelfl [lS] of experienced MEDLINE searchers, some relationships are noted between cognitive attributes and search process variables. In a study by Logan and Woelfl [19] of a small sample of novice searchers, initial results appeared to confirm findings from the MEDLINE study and suggest that the relationships between learning style and search behavior exist regardless of searcher experience. The current study of novice searchers explores the relationships between measures of cognitive learning style as deter- mined by three written tests and five measures of online performance. Since the focus of this study is the online process, outcome measures such as precision and recall are not addressed.

3.1 Tests 3.1.1 The learning style inventory (LSI). The LSI is a self-descriptive instrument

devised by David Kolb and based on work of Kurt Lewin during group dynamic studies conducted in the late 1940s [20], and used to measure preference for four basic modes of learning: concrete experience, reflective observation, abstract conceptualization, and active experimentation. From the preferences indicated for these learning modes, four raw scores and two composite scores determine placement on a Learning Style Grid and indicate one of four styles of learning: convergence, divergence, assimilation, or accommodation. A previous study has demonstrated a relationship between learning style and interactive database use explaining some variability in a task similar to online searching [22].

3.1.2 The remote associates test (RAT). The RAT is a test evaluated as being closely related to verbal reasoning 1231 and is used in Woelfl’s MEDLINE study as a measure of verbal inference, or the ability to infer a search strategy from a query.

3.1.3 The symbolic reasoning test (SRT). The SRT is taken from the Employee Apti- tude Survey [24] and measures the ability to draw correct conclusions from symbolic state- ments. In the MEDLINE study it is used as a measure of nonverbal reasoning, or the ability to perform Boolean operations, an essential online skill.

3.2 Search measures The measures provide an objective method of evaluating individual online behavior

during a search and include:

Cognitive styles and online behavior of novice searchers 505

Cycles the number of iterations during a search, generally indicated by the “type” command used to examine the results of a strategy; Commands the number of directives issued in response to the system prompt; Descriptors the number of actual terms for which the system is asked to search; Connect Time the total amount of time spent online during a search; References the number of records typed or printed out during a search.

3.3 Subjects The subjects of this study are 76 members of Florida State University’s Graduate

School of Library and Information Studies enrolled in the beginning online searching course. These students have had minimal exposure to the online process, but may have had microcomputer experience. Data has been gathered for the spring and fall terms over a three year period, 19851987.

3 A Methodology The tests of learning style, LSI, RAT, and SRT, are administered during the first class

period of the introductory online searching course. Tests are coded to preserve anonymity and are not evaluated until the course work is completed and grades assigned. To provide demographic information, students are asked for brief autobiographical information prior to administering the tests. The five measures of online searching behavior are applied to printouts from two searching assignments: a search for bibliographic information on two assigned topics, and a search initiated by the student from a client query. The two assigned topics remain the same throughout the study; the type of client query is not controlled. Nonnative speakers of English are eliminated from the sample. The database of all scores for all students is analysed for relationships using the MINITAB [25] statistical analysis program.

4. RESULTS

4.1 Learning style inventory Learning Style Inventory scores indicate a large percentage (45%) of the students fall

in the Diverger Quadrant, and a considerably smaller percentage (9Oro) fall in the Converger Quadrant. Divergers are characterized as operating well in situations that call for open ended generation of alternatives, as being creative thinkers, and having their greatest strength in imaginative ability. Convergers, on the other hand are best in situations where “right” answers can be found to a problem, are best at deductive reasoning, and often do well on I.Q. type tests. Placement in the Accommodator and Assimilator Quadrants is al- most evenly divided (22% and 24%, respectively). Accommodators are described as solving problems in intuitive, trial-and-error ways, adapting well to immediate and specific demands, and being best in situations that require action. Assimilators are best at inductive reasoning and assimilating disparate observations into integrated explanations. Fig- ure 1 shows the LSI Quadrant placement of FSU students.

Learning Style Grid

Active Reflective Experimentation Observation

Concrete Accomodator Diverger Experience 22% (17) 45% (34)

Abstract Converger Assimilator Conceptualization 9% (7) 24% (18)

Fig. 1. Quadrant placement of FSU students on the Learning Style Grid by percent and (number).

IPM 26:4-E

506 E. LOGAN

Table 1. Comparison of Learning Style Inventory Quadrant distribution of subjects from four studies

LSI Quadrant placement

ACC = Accommodator, ASS = Assimilator, CON = Converger, DIV = Diverger

FSU Saracevic N= 16 N= 36

Woelfl N= 35 Psvch

Borgman N = 62

Ene Enel

ACC 17 9 6 4 2 4 ASS 18 10 11 6 6 6 CON I 14 14 4 10 4 DIV 34 3 4 6 0 10

Comparing the FSU distribution with those of the original Woelfl study [26], the Borg- man study [27], and the Saracevic study [28], FSU students more closely resemble the English majors from the Borgman study, whereas the subjects from the Woelfl and Sara- cevic studies and the Borgman engineers have similar distributions. Table 1 gives these comparisons.

4.2 Remote associates test and symbolic reasoning test The mean scores of the RAT and SRT testsappear to be comparable among all four

groups, although FSU scores on the SRT are lower. Table 2 shows comparisons for the four studies.

4.3 Relationships of individual searching measures to test scores Using multiple regression techniques to examine relationships among the five measures

of online behavior and test scores, results show little correlation between individual measures. Correlating composite scores from the LSI with measures of online searching gives correlations well below ranges of statistical significance, although they exhibit a consistency which may be worth noting. Correlations between scores which reflect placement (see Fig. 1) on the continuum between abstract conceptualization and concrete experience (higher scores indicate an “abstract” rating) and the measures of online behavior show a consistently positive relationship. Conversely, correlations between scores indicating placement on the active experimentation and reflective observation continuum (higher scores indicate an “active” rating) are consistently negative. At the same time, although well below statistically significant figures, all but one RAT score correlation with online searching measures is negative and all the SRT score correlations are positive. Although these results are far from definitive, it does appear that both “active” LSI ratings and RAT scores demonstrate a weak negative relationship to the amount of activity online, whereas “abstract” LSI ratings and SRT scores show a weak positive relationship to online activity.

Table 2. Comparisons of RAT and SRT scores from four studies

Mean test scores for rat and SRT

FSU Saracevik Woelfl Borgman N= 76 N= 36 N= 35 N= 62

Mean S.D. Mean S.D. Mean S.D. Mean S.D.

RAT 13.04 4.6 13.03 NA 14.4 5.7 10.3 5.2 SRT 9.1 6.8 10.6 NA 11.7 6.0 14.7 6.0

Cognitive styles and online behavior of novice searchers

Table 3. FSU mean scores for online measures by LSI Quadrant. N = 76

Online mean scores by quadrant groups

507

ACC = Accommodator, ASS = Assimilator, CON = Converger, DIV = Diverger

ACC ASS CON DIV

Cycles Commands

3.32* 23.88’ 4.89; 32.59* 3.45* 27.21 3.74 25.41

Descriptors

20.26* 30.63’ 17.65’ 23.58

Connect Time

.321*

.467*

.340

.344

References

17.5’ 33.47+ 19.18* 23.19

*Statistically significant at p = c.05.

4.4 LSI grid placement and searching measures Looking at the relationships between the five measures of online behavior and place-

ment on the Learning Style Grid, the findings are more interesting. As can be seen in Fig. 1, subjects rating themselves high in the areas of active experimentation and concrete experience are placed in the Accomodator Quadrant. High ratings in reflective observation and abstract conceptualization place the subject in the Assimilator Quadrant. Placement in the other two quadrants follows the same pattern. Recall that accomodators are characterized as solving problems intuitively by trial and error and as being action oriented. Assimila- tors tend to use inductive reasoning, create theoretical models, and “assimilate” experiences into integrated explanations. Convergers use deductive reasoning to solve problems and they do well in tests where one correct answer is required. Divergers operate best in open- ended situations where many alternatives are desired. Mean scores for each of the quadrant groups are shown for the five searching measures in Table 3.

Assimilators as a group spend more time online, issue more commands, key more descriptors, complete more cycles, and print more references than any other group. Accom- modators, on the other hand have the lowest mean scores on all measures except descriptors. The Assimilator quadrant and the Accommodator quadrants are in opposite positions on the LSI Quadrant Grid. Assimilators rate themselves high in reflective observation and abstract conceptualization and can be considered more “relective” or “abstract.” Accom- modators rate themselves high in concrete experience and active experimentation and can be considered to be more “active” or “concrete.”

Individual scores for the five searching measures vary widely among searchers, an observation that is consistent with reports from other online studies. Minimum and maximum scores are given in Table 4.

Interestingly, all maximum scores are those of a single searcher and four of the five minimum scores are those of one other searcher. The “maximum” searcher has LSI scores that identify him as an assimilator; the “minimum” searcher is an accommodator.

Comparisons of mean RAT and SRT scores for quadrant groups appear to confirm consistency among tests of learning style. The Remote Associates Test is included as a measure of inductive reasoning; one of the characteristics of an assimilator is the ability to use inductive reasoning. The Symbolic Reasoning Test is included as a measure of deductive reasoning; convergers are characterized in part as using deductive reasoning. Mean RAT and SRT scores for quadrant groups are given in Table 5.

Table 4. FSU maximum and minimum scores for five measures of online process

Maximum and minimum scores for searching measures

MIN MAX

Cycles Comands

1 5 11 98

Descriptors Connect Time References

4.5 .057 1 101 .983 113

508 E. LOGAN

Table 5. FSU mean scores for RAT and SRT by LSI Quadrant. N = 76

RAT and SRT scores bv auadrant ~rouos

Accommodator Assimilator Converger Diverger

RAT 13.61 14.47+ 10.3 12.75 SRT 9.11 10.63 11.12* 8.15

*Statistically significant at p = < .05.

4.5 Comparisons with other studies Comparisons between the results of this investigation and the Saracevic and Woelfl

studies show both confirmations and contradictions. Mean scores for quadrant types for three studies are shown in Table 6.

Both the FSU and the Saracevic studies show the highest mean scores for all five variables occurring in the Assimilator Quadrant. Subjects in the Woelfl study with the highest mean scores are all in the Diverger Quadrant. Since high mean scores indicate more activity and thus more time online, it is interesting that in two of the three studies, highest mean scores are associated with a single quadrant group. However, the lowest mean scores, indicating less activity online and less time spent are not as consistent. In the Sara- cevic study, the majority of low scorers are divergers (the very ones who scored highest in the Woelfl study), those in the FSU study are accommodators, and those in the Woelfl study are divided between assimilators (highest in the FSU are Saracevic studies) and accommodators (lowest in the FSU study).

Individual high and low scores appear to have no consistent relationships either within the studies or when compared with the study at FSU. In the Saracevic study, two divergers, an accommodator and a converger are the low scoring subjects; in the Woelfl study, low scores are shown by two accommodators, a converger, and an assimilator. High scores in the Saracevic study are shown by two assimilators and two convergers; in the Woelfl study, high scorers are two assimilators, two accommodators, and a diverger. Note, however, that in all three studies, assimilators are associated at least in part, with the higher scores.

Table 6. Mean scores of online measures for FSU N = 76, Saracevic N = 36, (SRV), and Woelfl N = 35, (WFL) studies

Mean Score Comparisons

Cvcles Commands Descriotors Connect Time References

ACC FSU

SRV WFL

ASS FSU

SRV WFL

CON FSU

SRV WFL

DIV FSU SRV WFL

3.32 23.88 20.26 .321

3.62 15.41 8.89 .257 2.58 22.50 12.83 .338

4.89 32.59 30.63 .467

3.65 17.64 11.28 .292 2.39 25.72 11.55 .322

3.45 27.21 17.65 .340

3.51 16.55 8.08 .209 2.45 28.18 12.02 ,321

3.14 25.41 23.58 .344 2.10 8.57 8.11 .175 2.63 40.13 14.13 .365

17.15

2:.al3

33.47

3:;s

19.18

3o?a32

23.19 na

49.50

Cognitive styles and online behavior of novice searchers

5. CONCLUSIONS AND DISCUSSION

509

Results from the FSU study show some interesting correlations between LSI measures and online performance. Assimilators show higher mean scores on all five searching measures; accommodators show lower mean scores on four of five. Since assimilators are those who rate high in reflective observation and abstract conceptualization, and accommodators rate high in active experimentation and concrete experience, they occupy opposing quadrants in the Learning Style Grid. It would appear that assimilators and accommodators also demonstrate equally opposing modes of behavior online. In addition, the subject with the highest individual scores in online processes identifies himself as an assimilator; the subject with the lowest scores identifies herself as an accommodator. These results appear to suggest that subjects are able to evaluate themselves on the LSI in a manner consistent with their behavior when online.

This consistency between LSI quadrant placement and online styles suggests the possibility of determining likely searcher behavior prior to online performance. The Learn- ing Style Inventory is a simple twelve-item test in which subjects rank their preferences for modes of learning. If further studies were to confirm these findings, the LSI or an equiv- alent could assist in administrative decisions, selection of staff, choice of training meth- ods, and assignment of searching duties. In the larger context of developing automated systems for searcher assistance, learning styles may be a factor worth considering in heuristics for expert systems, for menu driven selection tools, and other automated end user pro- grams [29].

Comparing the FSU study with results from the Saracevic and Woelfl studies, the results are less conclusive. Only in the Saracevic study is the performance of the assimilators duplicated with the same consistency. In the Woelfl study, higher mean scores of online performance are not associated with assimilators. Comparing lower mean scores among the studies, results appear even less consistent. While there seems to be some con- firming evidence (both FSU and the Saracevic study show high mean scores associated with assimilators and both FSU and Woelfl show some low mean scores for accomodators), the results of the FSU study are only partially confirmed. In part, the discrepancy may be explained in terms of the influence of increased experience and training of the searchers in the Saracevic and Woelfl populations. Since novice searchers have little online training or experience, they may rely more on their basic learning styles to guide their behavior when online. Experienced searchers may rely less on learning styles and more on acquired techniques. Additional research is needed to address this possibility.

The partial confirmation of the FSU results by the Saracevic and Woelfl studies suggests a need for further explorations in this area and, unfortunately, an admission that there is still no conclusive evidence for documenting the characteristics associated with online behavior. Individual differences in online search unexplained.

1.

2.

3.

4. 5.

6.

7.

8.

REFERENCES

process, for the most part, remain

Williams, Martha E. Transparent information systems through gateways, front ends, intermediaries, and interfaces. J. Amer. Sot. Info. Sci., 37: 204-214; 1987. Saracevic, Tefko, et al. A study of information seeking and retrieving. I. background and methodology. J. Amer. Sot. Info. Sci., 39: 161-176; 1988. Kraft, Donald H. Advances in information retrieval: Where is that /#*%@ 1/4 record? Adv. in Comp., 24: 277-318; 1985. Gordon, Sallie E. The human factor in expert systems. AI Expert, 3: 55-60; 1988. Croft, W.B.; Thompson, R.H. I’R: A new approach to the design of document retrieval. J. Amer. Sot. Info. Sci., 38: 389-404; 1987. Swanson, Don R. Historical note: Information retrieval and the future of an illusion. J. Amer. Sot. Info. Sci., 39: 92-98; 1988. Belkin, Nicholas J. et al. Distributed expert-based information systems: An interdisciplinary approach. Info. Proc. & Man., 23: 395-409; 1987. Swanson, Don R. Historical note: Information retrieval and the future of an illusion. J. Amer. Sot. Info. Sci., 39: 92-98; 1988.

510 E. LOGAN

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21. 22.

23.

24.

25.

26.

27.

28.

29.

Croft, W.B.; Thompson, R.H. 13R: A new approach to the design of document retrieval systems. J. Amer. Sot. Info. Sci., 38: 389-404; 1987. Woelfl, Nancy N. Individual differences in online bibliographic searching: The effect of learning styles and cognitive abilities on process and outcome. Ph.D. Dissertation. Cleveland, OH: Case Western Reserve Uni- versity; 1984. Saracevic, Tefko; Kantor, Paul. A study of information seeking and retrieving. III. Searches, searchers, and overlau. J. Amer. Sot. Info. Sci., 39: 197-216; 1988. Fenichel, Carol. Online searching measures that discriminate among users with different types of experience. J. Amer. Sot. Info. Sci., 23: 23-32; 1981. Bellardo, Trudi. An investigation of online searcher traits and their relationship to search outcome. J. Amer. Sot. Info. Sci., 36: 241-250; 1985. Fidel, Raya. Online searching styles: A case-study-based model of searching behavior. J. Amer. Sot. Info. Sci., 35: 211-221; 1984. Brindle, E.A. The relationship between characteristics of searchers and their behavior while using an online interactive retrieval system. Ph.D. Dissertation. Syracuse, New York: Syracuse University; 1981. Borgman, Christine L. Individual differences in the use of technology: Work in progress. Proc. Amer. Sot. Info. Sci., 22: 243-249; 1986. Nilan, Michael; Peck, Robin P.; Snyder, Herbert W. A methodology for tapping user evaluation behaviors: An exploration of users’ strategy, source, and information evaluating. Proc. Amer. Sot. Info. Sci., 25: 152- 159; 1988. Woelfl, Nancy N. Individual differences in online bibliographic searching: The effect of learning styles and cognitive abilities on process and outcome. Ph.D. Dissertation. Cleveland, OH: Case Western Reserve Uni- versity; 1984. Logan, E.L.; Woelfl, Nancy N. Individual differences in online searching behavior of novice searchers. Proc. Amer. Sot. Info. Sci., 23: 163-165; 1986. Kolb, David A. Experiential learning theory: Experience as source of learning and development. Englewood Cliffs, N.J.: Prentice-Hall; 1973. Kolb, David A. The learning style inventory. Boston: MC Ber and Company, 1981; 1985. Grochow, Jerrold. Cognitive style as a factor in the design of interactive decision support systems. Ph.D. Dissertation; Boston, Mass; Massachusetts Institute of Technology; 1973. Baird, Leonard L. The remote associates test. In: Oscar K. Buros editor. The seventh Mental Measurements Yearbook. Highland Park, N.J.: Gryphon Press; 1972. Ruth, Floyd L.; Ruth, William L. The employee aptitude survey. Los Angeles, CA: Psychological Services, Inc.; 1957. MINITAB is a statistical package written in FORTRAN IV and available from The Pennsylvania State Uni- versity, University Park, PA. Woelfl, Nancy N. Individual differences in online bibliographic searching: The effect of learning styles and cognitive abilities on process and outcome. Ph.D. Dissertation. Cleveland, OH: Case Western Reserve Uni- versity, 1984. Borgman, Christine. Individual differences in the use of technology: Work in progress. Proc. of the Amer. Sot. for Info. Sci., 22: 243-249; 1986. Saracevic, Tefko: Kantor, Paul. A study of information seeking and retrieving. III. Searches, searchers, and overlap. J. Amer. Sot. Info. Sci., 39: 197-216; 1988. Dr. Wyllis Bandler of the Computer Science Department at Florida State University is investigating the influence of learning styles on decision-making in city planning for an expert system design.

cognitive styles and online behavior of novice searchers

Documents