Analyzing Skill Sets With or-Relation Tables in Knowledge Spaces

Feifei xu, Duoqian MiaoDepartment ofComputer Science and Technology, Tongji University, Shanghai, 201804 China

xufeifei1983@hotmail.com,miaoduoqian@163.comYiyu Yao

Department ofComputer Science, University ofRegina, Regina, Saskatchewan, Canada S4S OA2yyao@cs. uregina. ca

Lai WeiDepartment ofComputer Science and Technology, Shanghai Maritime University, Shanghai

200135 Chinaweily105@hotmail.com

Abstract

The disjunctive model ofskill map in knowledge spacescan be interpreted based on an or-binary relationtable between skills and questions. There may existskills that are the union ofother skills. Omitting theseskills will not change the knowledge structure. Findinga minimal skill set may be formulated similar to theproblem of attribute reduction in rough set theory,where an and-binary relation table is used. In thispaper, an or-relation skill-question table is consideredfor a disjunctive model of knowledge spaces. Aminimal skill set is defined and an algorithm forfinding the minimal skill set is proposed. An example isused to illustrate the basic idea.

1. Introduction

Cognitive science [16][18] is an interdisciplinarystudy related to psychology and artificial intelligence.Its scope covers a wide range of topics, includingknowledge acquisition and representation. In fact, amain task ofcognition is knowledge acquisition [1].

Cognitive informatics, initiated by Wang and hiscolleagues [22-28], is born from the marriage ofcognitive and information sciences. It investigates intothe internal information processing mechanisms andprocesses of the natural intelligence (i.e., human brainsand minds) and artificial intelligence (i.e., machines).An important topic in cognitive informatics is thestudy of knowledge.

Prlc.lib IEEE lIt Cllf.II Cillidlillflr.adcs (ICCI'OI)I. Bacil, I. Wall, 1.1.lal, W.IiISllr, I.Chal& L.A. Zadlh (Eds.)111-1-4244-4642-1/01/825.00 ©20011EEE

174

The theory of knowledge spaces [2-5] represents anew paradigm in mathematical psychology forknowledge assessment. Students' knowledge states arerepresented and assessed systematically by using afinite set of questions. A collection of subsets ofquestions is called a knowledge space in which eachsubset is called a knowledge state or a cognitive state.The family of knowledge states may be determined bythe dependency of questions or the mastery of differentsets of questions by a group of students. To beconsistent with the traditional explanatory features ofpsychometric theory, the notion of skills may be usedto interpret students' knowledge states. Specifically, aquestion can be described by several skills.

In many situations, students' knowledge states canbe inferred from data sets that summarize students'ability to answer certain questions. Binary relationtables are used as a common tool to describe the datasets. For such a table, it is assumed that rows denoteobjects and columns denote attributes of the objects.When the dimensionality of the data set is huge, it maybe difficult to find rules and discover usefulknowledge from it. One needs to reduce the table, ifpossible, without loss of the valuable information. Asuccessful method to deal with such a problem isattribution reduction in the rough set theory.

Rough set theory [12], proposed by Pawlak in 1982,is useful for discovering knowledge hidden in a dataset. One of the significant contributions of rough settheory is attribute reduction. Typically, there may existsome attributes that do not provide additionalinformation and these redundant attributes can beremoved. The notion of a reduct is a minimal subset ofthe attributes that provides the same information as the

entire set of attributes [12]. Through the attributereduction, we can work with a reduced informationtable in order to find rules efficiently. Attributereduction may be viewed as a granular computingapproach. It reduces the redundant attributes in thetable while keeping the granularity of knowledge.

The relationship among the attributes in aninformation table or a decision table is traditionallyinterpreted as an and-relation. That is, an object mustsatisfy the values of all attributes at the same time.Rough set researchers focus on this type of and­relation tables [6-11][17][19-21][29-30][32-40]. In thispaper, we consider an alternative interpretation ofbinary relation tables. The relationship among theattributes is interpreted as an or-relation. That is, anyone of the attributes is sufficient to describe an object.This kind of binary relation tables is useful inknowledge spaces.

There are two kinds of skill maps in knowledgespaces. One is the disjunctive model, in which aquestion can be answered if anyone of a set of skills ismastered. For instance, if one wants to calculate thevalue of the expression, 2 +2 +2 +2 +2 =2 x 5,one only needs to master of the skills in the set{addition, multiplication}. The other model is theconjunctive model, in which a question can beanswered by mastery of all skills in a set of skills. Forexample, one can only correctly calculate the value of2 x 5 - 1 if one to masters both skills in the set{multiplication, subtraction}. Skills in the disjunctivemodel are modeled by an or-relation, and skills in theconjunctive model are modeled by an and-relation.This paper focuses on discussing the or-relation tables.

We can set up a skill-question table where rows arequestions and columns are skills. That is, questions areconsidered as objects and skills are attributes. Ifsolving a question needs to master some skills, wedenote the corresponding skill value as 1 and 0otherwise. From such a skill-question table, thedependencies of questions can be obtained. In thedisjunctive model, for each skill, we can get a questionset, i.e., a knowledge state. For an arbitrarycombination of skills, some knowledge states areobtained. All knowledge states, together with theempty set and the entire question set, form aknowledge structure. It has been proved that such aknowledge structure is closed under set union, which isnamed as a knowledge space [3].

In the disjunctive model, it may happen that oneskill is the union of other skills in the table. That is, theskill is redundant and removing it will not cause anychanges in the derived knowledge structure. A largenumber of such superfluous skills will lower the

175

efficiency of building, testing and searching for aknowledge space and waste the storage space. It istherefore useful to find a minimal skill set in a skill­question table. We adopt some notions from the roughset theory in designing such an algorithm for finding aminimal skill set that has the same power forconstructing a knowledge space.

Knowledge spaces [2-5] and rough sets [12-15] aretwo related theories. One can adopt ideas from one inattempt to enrich the other, as well as combining ideasfrom both theories. Rough set approximations havebeen introduced into knowledge spaces in our previouswork [31]. In this paper, we adopt the ideas of attributereduction from rough set theory to knowledge space.This not only leads to more insights into attributereduction, but also brings us closer to a commonframework for studying the two related theories.

Knowledge representation and processing isimportant to cognitive computing and cognitiveinformatics. The study of knowledge spaces istherefore relevant to cognitive informatics. The rest ofpaper is organized as follows. Section 2 presents somerelevant background about knowledge spaces. Section3 gives an algorithm for finding a minimal skill set.Section 4 summarizes the main results of this study.

2. Overview of Knowledge Spaces

This section presents an overview of knowledgespaces by introducing its basic concepts and notions[2-5].

2.1. Knowledge spaces

The theory of knowledge spaces was proposed andstudied by Doignon and Falmagne [2-5] in 1985. Itstarts out with rather simple psychological assumptionson assessing students' knowledge based on their abilityto answer questions. Knowledge spaces may be viewedas a theory of information presentation andinformation use. The main objective of knowledgespaces is to effectively and economically solve theproblem of knowledge assessment. Many researchershave contributed to the theory and severalcomputerized procedures have been implemented [2-5].

In knowledge spaces, one uses a finite set ofuniverse (i.e., a set of questions denoted by Q) and a

collection of subsets of the universe (i.e., a knowledgestructure denoted by K), where K contains at least theempty set 0 and the whole set Q. The members of

K are called the knowledge states or cognitive statesthat are subsets of questions given by experts orcorrectly answered by students.

There are two types of knowledge structures. One isthe knowledge structure associated to a surmiserelation, closed under set union and intersection. Theother is the knowledge structure associated to asurmise system, closed only under set union. From theview of granular computing, a knowledge state can bereferred to as a granule. The knowledge structure maybe viewed as a granular structure.

A sunnise relation on the set Q of questions is a

transitive and reflexive relation S on Q. By aSb,we can surmise that the mastery of a if a student can

answer correctly question b. This relation imposescertain conditions on the corresponding knowledgestructure. For example, sunnising the mastery ofquestion a if a student can answer correctly question

b means that if a knowledge state contains b, it must

also contain a. The knowledge structure associated toa sunnise relation can be formally defined [2]:

Definition 1. For a surmise relation S on the

(finite) set Q of questions, the associated knowledge

structure K is defined by:

(( = {KI(Vq,q' E Q,qSq',q' E K)~ q E K},(1)

where K is a knowledge state.With surmise relations, a question can only have

one prerequisite. Sometimes, this may be toorestrictive. In practice, we may assume that aknowledge structure is closed only under union, calleda knowledge space. A knowledge space is a weakenedknowledge structure associated to a surmise relation.

We can define a knowledge space with a surmisesystem. A surmise system on a (finite) set Q is a

mapping a that associates to any element q in Q. A

nonempty collection a(q) of the subsets of Qsatisfies the following three conditions:1) Cea(q)=:>qeC;

2)(C e a(q),q' e C) =:> (3C' e a(q'),C' ~ C);

3) C e a(q) =:> (VC' e a(q),C' a: C).

The subsets C, C' in a(q) are the clauses for

question q. The corresponding knowledge structure

has the following definition.

176

Definition 2. For a surmise system (Q, a ) , the

associated knowledge structure is defined by:

K = {KI(Vq e Q,q e K)~ (C e a(q),C e K)}.(2)

They constitute the knowledge structure associated

to (Q, a). In fact, there is a one-to-one

correspondence between surmise systems on Q and

knowledge spaces on Q.There are several approaches to obtain a surmise

relation or a surmise system. From the existingresearch, we can identify at least the followingmethods:

1. Querying experts;2. A posteriori analysis ofmass data;3. Analysis ofdidactics and curricula;4. Systematical problem construction;5. Analysis of skills, demands, competence (latent

structures).

A preferable method may be the analysis of skillsand competence. Many researchers concentrate on thiskind of sub-theory. We assume that the surmiserelation or the surmise system is obtained by the skills.

2.2. Skill-question tables in knowledge spaces

Given a set of questions Q, the more skills one has,

the more questions one is capable of solving. Thissimple idea is the basis for studying the connectionbetween skills and questions.

Given a set S of skills, for each subset Y of S,there exists a set of questions that can be solved by theskills in Y. With such a connection, it isstraightforward to compute all possible knowledgestates.

To establish the relation between skills andquestions more intuitively, we can use a binary relationtable. The rows represent the questions and thecolumns represent the skills. Mastery of a questionneeds one to have some skills; we set 1 as the values inthe corresponding rows and columns. Otherwise, thevalues are assumed to be o. Table 1 is an example of abinary relation table.

With respect to the different meanings of skills,there are two models. A question may be solved if anyone skill is mastered. That is, the relationship betweenskills is an or-relation. Mastery of anyone skill iscapable of solving the question. It is called thedisjunctive model. Alternatively, solving a question

needs the mastery of all the skills in a subset of skills.It is called the conjunctive model. In the conjunctivemodel, the relationship between skills is an and­relation that is the same as an information table in therough set theory.

This paper uses the idea of attribute reduction inrough set theory. We mainly discusses how to find aminimal skill set to save the storage space and increaseof efficiency of knowledge assessment, while keepingthe corresponding knowledge structure in an or­relation table.

Table 1. A Skill-Question Table51 52 53 54 55

a 1 1 0 1 0b 1 1 1 0 0c 0 1 1 0 0d 1 1 0 0 1e 0 0 0 0 1

3. The Minimal Skill Set in the DisjunctiveModel in Knowledge Spaces

The notion of a minimal skill set is introduced andan algorithm for finding such a set is given.

3.1. Minimal skill set

Let Q be a nonempty set of questions and S be a

nonempty set of skills. Suppose T is a skill mapping

from Q to 2s\ {0}. For any q in Q, the subset

T(q) of S is referred to as the set of skills assigned

to q (by the skill map T).

Suppose Y is a subset of S. We say that K c Qis the knowledge state delineated by Y via thedisjunctive model if

K={qeQlr(q)nY:;t:0}. (3)

From the Table 1, the knowledge state delineated by

Y ={SI,S5} is {a,b,d,e}. The subset {b,d} is

not a knowledge state, since it cannot be delineated byany subset Y of S. The knowledge space delineatedfor Table 1 is given by:fl ={0, {a},{b,c},{d,e},{a,b,d}, {a,b,c},{a,d,e},

{b,c,d,e},{a,b,c,d},{a,b,d,e},{a,b,c,d,e}}.(4)

177

In the disjunctive model, a skill may be composedof other skills. In a skill-question table, it is shown thatthe question subset delineated by a skill can be theunion of other question subsets delineated by other

skills. Using Table 1, S2 consists of Sl and S3' since

the knowledge state delineated by S2 equals to the

union of the knowledge states delineated respectively

by Sl and S3. That is, S2 can be represented by Sl

and S3. We only need to leave one skill subset

{SI,S3,S4,S5}. We say that S2 is redundant in

computing a knowledge space, because the knowledge

state delineated by S2 is the same as the knowledge

state delineated by {Sl' S3 }. It can be easily shown

that the knowledge space delineated by {SI,S3,S4,S5}

is the same as the knowledge space delineated by theoriginal skill-question table, but not the same as that

delineated by {S2,S4,S5}.

Redundant skills limit the effective and economicassessment of knowledge and waste memory. For theefficient processes, we need to find a minimal skill.First, we state the properties that must be satisfied by aminimal skill set in a skill-question table.

Definition 3. A minimal skill set of a skill-questiontable should satisfy two conditions:a) The knowledge space delineated by the decreased

skill-question table should be the same with thatdelineated by the initial table;

b) The knowledge state delineated by a single skillcannot be the union of the knowledge statesdelineated by other skills.

It follows that a minimal skill set of a skill-questiontable are the most compact skill-question table whichcan delineate the same knowledge space as the originaltable. We can also determine a skill whether it can bereplaced by the other skills

Definition 4. A skill that can be removed mustsatisfy two conditions:a) The knowledge space delineated by the skill­

question table will not change after omitting thecolumn of the skill;

b) The knowledge state delineated by the skill is theunion of the knowledge states delineated by otherskills.

3.2. Algorithm for finding a minimal skill set of granular computing, it maintains the granularity ofthe knowledge space.

4. Conclusion

The knowledge space delineated by the decreasedtable 2 is given as follows:

K={0, {a},{b,c}, {d,e},{a,b,d},{a,b,c},{a,d,e},

{b,c,d,e},{a,b,c,d},{a,b,d,e},{a,b,c,d,e}}

which is the same with the knowledge space delineatedby Table 1.

3.3. An example

Based on Table 1, the skill set S ={SI'S2' ...' S5} ,and the family of the associated question sets is{d(s;) Ii= 1,2,...,5} = {{a,b,d}, {a,b,c,d} ,{b,c} ,{a} ,{d,e}}.

The family of Ds; for each skill is

{DSiIi=1,2,...,5} ={{d(S4)}' {d(SI)' d(S3),d(S4)}' {0}}.

It is easy to show that d(S2) = UDS2

• Therefore, S2

can be removed. The decreased skill-question table isshown in Table 2:

We present a new interpretation of a skill-questiontable under the disjunctive model in knowledge spaces.Similar to the rough set theory, skills can be viewed asthe attributes and questions as objects. However, therelationship between attributes is no longer an and­relation but or-relation. That is, a question can besolved by mastery of only one skill. An algorithm forfinding a minimal skill set in an or-relation table isproposed.

Or-relation tables are a new type of tables. A skill­question table in the disjunctive model can beconsidered as a special case in or-relation tables. Thispaper demonstrates some potential value on studyingor-relation tables.

Approximations and reduction are two of thecentral topics in the rough set theory. We introducethem into knowledge spaces. The results from this

e2. A Decreased Skill-Question Tablsl s3 s4 s5

a 1 0 1 0b 1 1 0 0c 0 1 0 0d 1 0 0 1e 0 0 0 1

Table

Algorithm: Finding a minimal skill set

Definition 5. The core skill of a skill set is definedby:

CORE(S) = {s; IDs, = {0},l::;; i::;; m}. (6)

Base on these notations, we give an algorithm forfinding a minimal skill set.

By using the algorithm, we can remove theredundant skills from a skill-question table of thedisjunctive model. Compared to the attribute reductionin the rough set theory, finding a minimal skill set canalso be regarded as reducing the redundant informationwhile keeping the knowledge structure. From the view

From a skill-question table, we can obtain thequestion set for each skill. Suppose we have a set of

skills S ={SI'S2' ...,Sm}. Then all the questions that

skill Sj (1 :::; i :::; m) can solve constitute the question

set d(sj). We denote D s; as the collection of the

question sets for some skill contained in d (s,).Namely

Ds, = {d(Sk)ld(sk) c d(s),l::;; k::;; m.k » i}.(5)

With the definition above, we can see that a skill

can be removed if it satisfies d(sj) = UDs; . A skill

which can not be removed should satisfy D s; ={0}.We can define a subset of skills consisting of all the

skills satisfying D s; ={0} as the core skill of the

skill set.

STEP 1: For each skill, the corresponding question setis obtained. Namely, for the skill set

S ={SI'S2' ..., Sm} , find the question sets

{d(SI)' d(S2)' ...,d(sm)};

STEP 2: Compute every skill's Ds; ' namely,

{Ds1' DS2 ' ••• , DSm };

STEP 3: For each skill Sj E S, remove it from S

ifd(sj) =UDs; ;

STEP 4: Output the rest skill subset R =S.

178

study show a strong connection between the tworelated theories. As future work, we plan to investigatesystematically the connections between rough settheory and knowledge spaces based on a commonframework of granular computing.

Acknowledgement

The research is supported by the National NaturalScience Foundation of China under grant No:60775036, 60475019, and the Research Fund for theDoctoral Program of Higher Education of China undergrant No: 20060247039.

References

[1] Chiew, v.: A Software Engineering CognitiveKnowledge Discovery Framework. In Proceedings ofthe 1st IEEE International Conference on CognitiveInformatics, 163-172, 2002

[2] Doignon J.P., Falmagne J.C.: Spaces for TheAssessment of Knowledge. International Journal ofMan-Machine Studies, 23, 175-196, 1985.

[3] Doignon, J.P., Falmagne, J.C.: Knowledge Spaces.Springer, 1999.

[4] Duntsch, I., Gediga, G.: A Note on the Correspondencesamong Entail Relations, Rough Set Dependencies, andLogical Consequence. Mathematical Psychology, 43,393-401,2001.

[5] Falmagne, J.C., Koppen, M., Villano, M., Doignon, J.P.,Johanessen, L.: Introduction to Knowledge Spaces:How to Test and Search Them. Psychological Review,97,201-224, 1990.

[6] Hu, Q.H., Xie, Z.X., Yu, D.R.: Hybrid AttributeReduction Based on A Novel Fuzzy-rough Model andInformation Granulation. Pattern Recognition, 40,3509-3521, 2007.

[7] Liang, H.L., Wang, J., Yao, Y.Y.: User-oriented FeatureSelection for Machine Learning. The Computer Journal,50(4),421-434,2007.

[8] Liang, J.Y., Xu, Z.B.: The Algorithm on KnowledgeReduction in Incomplete Information Systems.International Journal of Uncertainty, Fuzziness andKnowledge-Based Systems, 10(1), 95-103, 2002.

[9] Mi, J.S., Wu, W.Z., Zhang, W.X.: Approaches toApproximation Reducts in Inconsistent Decision Tables.In Proceeding of Rough Sets, Fuzzy Sets, GranularComputing, 283-286, 2003.

[10] Miao, D.Q., Hu, G.R.: A Heuristic Algorithm forReduction of Knowledge. Journal of ComputerResearch and Development, 36, 681-684, 1999. (inChinese)

[11] Nguyen, H.S.: On the Decision Table with MaximalNumber of Reducts. Electronic Notes in TheoreticalComputer Science, 82(4), 198-205,2003.

[12] Pawlak, Z.: Rough Sets. International Journal ofComputer and Information Sciences, 11,341-356, 1982.

179

[13] Pwlak, Z.: Rough Classification. International JournalofMan-machine Studies, 20, 469-483,1984.

[14] Pawlak, Z., Skowron, A.: Rough Sets: Some Extensions.Information Sciences, 177,28-40,2007.

[15] Polkowski, L.: Rough Sets: Mathematical Foundations.Advances in Soft Computing, Physica, Heidelberg, 2002.

[16] Posner,M.I.: Foundations of Cognitive Science, The ITPress, Cambridge, Massachusetts, 1989.

[17] Shen, Q., Chouchoulas, A.: A Modular Approach toGenerating Fuzzy Rules with Reduced Attributes for theMonitoring of Complex Systems. Artifical Intelligence,13,263-278, 2000.

[18] Simon, H.A., Kaplan, C.A: Foundations of CognitiveScience. Foundations ofCognitive Science, 1-47, 1989.

[19] Swiniarski, R.W.: Rough Sets Methods in FeatureReduction and Classification. International Journal ofApplied Mathematics and Computer Science, 11, 565­582,2001.

[20] Wang, G.Y.: Rough Reduction in Algebra View andInformation View. International Journal of IntelligentSystems, 18(6),679-688,2003.

[21] Wang, J., Miao, D.Q.: Analysis on Attribute ReductionStrategies of Rough Set. Chinese Journal of ComputerScience and Technology, 13(2), 189-192, 1998.

[22] Wang, Y.X.: Cognitive Models of the Brain. InProceedings of the 1st IEEE International Conferenceon Cognitive Informatics, 259-269, 2002.

[23] Wang, Y.X.: On Cognitive Informatics. In Proceedingsof the 1st IEEE International Conference on CognitiveInformatics (ICCI'02), 34-42, 2002.

[24] Wang, Y.X.: Cognitive Informatics: A NewTransdisciplinary Research Field, Brain and Mind. ATransdiciplinary Journal of Neuroscience andNeurophilosophy, 4(2), 2003.

[25] Wang, Y.X.: On Cognitive Informatics. Brain and Mind:A Transdiciplinary Journal of Neuroscience andNeurophilosophy, 4(2), 151-167,2003.

[26] Wang, Y.X., Gafurov, D.: The Cognitive Process ofComprehension. In Proceedings of the 2nd IEEEInternational Conference on Cognitive Informatics(ICCI'03), 2003.

[27] Wang, Y.X., Kinsner, W.: Recent Advances inCognitive Informatics. IEEE Transactions on Systems,Man, and Cybernetics, 36(2), 121-123,2006.

[28] Wang, Y.X.: The Theoretical Framework of CognitiveInformatics. International Journal of CognitiveInformatics and Natural Intelligence, 1(1), 1-27,2007.

[29] Wu, W.Z.: Attribute Reduction Based on EvidenceTheory in Incomplete Decision Systems. InformationScience. 178(5),1355-1371,2008.

[30] Xu, F.F., Miao, D.Q., Wei, L.: An Approach for Fuzzy­Rough Attribute Reduction via Mutual Information. InProceeding of The 4th International Conference onFuzzy Systems and Knowledge Discovery (FSKD'07),107-112,2007.

[31] Xu, F.F., Yao, Y.Y., Miao, D.Q.: Rough SetApproximations in Formal Concept Analysis andKnowledge Spaces. In The 17th International

Symposium on Methodologies for Intelligent Systems(ISMIS'08), LNAI4994: 319-328, 2008.

[32] Yao, Y.Y.: Information Retrieval Support Systems, InThe 2002 IEEE World Congress on ComputationalIntelligence, 773-778,2002

[33] Yao, Y.Y., Zhao, Y., Wang, J.: On Reduct ConstructionAlgorithms, In Proceeding of the 1st internationalConference on Rough Sets and Knowledge Technology,RSKT 2006, LNAI4062: 297-304,2006.

[34] Yao, Y.Y., Zhao, Y., Wang, J., Han, S.Q.: A Model ofUser-Oriented Reduct Construction for MachineLearning. Transaction on Rough Sets 8: 332-351, 2008.

[35] Ye, D.Y., Chen, Z.J.: An Improved DiscernibilityMatrix for Computing All Reducts of An InconsistentDecision Table. In the Proceedings of the 5th IEEEInternational Conference on Cognitive Informatics.305-308, 2006.

[36] Yu, H., Wang, G.Y., Lan, F.K.: Solving the AttributeReduction Problem with Ant Colony Optimization. In

180

Proceeding of Rough Sets and Current Trends inComputing, 242-251, 2008.

[37] Zhang, W.X., Mi, J.S., Wu, W.Z.: Approaches toKnowledge Reductions in Inconsistent Systems.International Journal ofIntelligent Systems. 18(9), 989­1000,2003.

[38] Zhao, K., Wang, J.: A Reduction Algorithm MeetingUsers' Requirements. Journal of Computer Science andTechnology. 17(5),578-593,2002.

[39] Zhao, Y., Luo, F., Wong, S.K.M., Yao, Y.Y.: A GeneralDefinition of An Attribute Reduct. In Proceeding of the2nd international Conference on Rough Sets andKnowledge Technology, RSKT 2007, LNAI4481: 101­108,2007.

[40] Zhu W., Wang, F.Y.: Reduction and Axiomization ofCovering Generalized Rough Sets. Information Science.152, 217-230, 2003