Integrating Decision Support Systems: Expert, Group,

and Collective Intelligence

Stephen R. Diasio1, Núria Agell2 1Department of Quantitative Methods, ESADE Business School, Barcelona, Spain 2Department of Quantitative Methods, ESADE Business School, Barcelona, Spain

Abstract - Organizations today face a changing environment,

where external conditions change rapidly, organizational

structures are more flat and dispersed, and where the

traditional roles of experts have been “squeezed” or of

decreased importance. These converging factors have

importance for organizations’ ability to remain competitive.

These evolving trends present challenges and opportunities

for organizations in which information technology will play a

vital role in supporting and facilitating decision-making. The

paper aims to understand how integration of expert systems,

group decision support systems, and collective intelligence

tools can enhance decision-making.

Keywords: Decision Support Systems, Knowledge Representation, Collective Intelligence, Artificial Intelligence

1 Introduction

Organizations today face a changing environment, where external conditions change rapidly [1], organizational structures are more flat and dispersed [2], and where the traditional roles of experts have been “squeezed” or of decreased importance [3]. As a result of these converging forces renewed interest in decision-making technology that enables organizations to survive and remain competitive has emerged. Though faced with a daunting task, information technology [4] continues to play an increasing role in facilitating a firm’s success. Researchers have argued [4] information technology is a woven thread in the fabric of the organization and affects both human behavior and organizational structure. Additionally, investigation has emphasized [5] today’s new environment places a premium on collaboration while the old organizational structure impedes advancement and is a detriment to the future existence of the organization. Trends and new developments in technology present challenges and opportunities for researchers within the fields of artificial intelligence (AI) and decision support technology. To understand this relationship, this paper focuses on expertise that is supported by expert systems (ESs), group decision support systems (GDSSs), and collective intelligence tools (CI Tools) to identify the potential benefits of integration of these systems for enhanced decision-making. A conceptual

framework outlining a newly fused system that enhances deliberation of complex problems will be presented. A literature review of each research areas’ characteristics and functions will be highlighted that allow organizations to leverage decision-making tools for effective decision-making. The paper is outlined as follows: a framework for integration and discussion how expertise is supported by technology in Section 2. Section 3 presents three decision support technologies that organizations use supporting expertise. Section 4 explores opportunities for enhancing decision-making and collective intelligence tools through integration of existing methodologies and AI techniques. Conclusions and direction for further research appear in Section 5.

2 Framework for Integration

Organizations and their leaders have an abundance of decision support tools at their disposal [6] to help achieve organizational goals. Historically, these tools have been independently built [7] to address problems businesses face but must remain flexible in order to adapt to meet the changing conditions and needs of the users [8]. Existing literature has individually shown the advantages of using such systems in addressing problems, however have not extended or offered in theory or practice an integrated system that supports organizational needs in expertise and in decision-making. A conceptual map (Figure 1) for a proposed integrated support system is presented for research and development. Throughout all levels of organizations decisions are made requiring a wide continuum or information. Many decisions are dependent on the available information at hand or able to be collected. Decision-making can be compromised if decision-makers do not have access to the resources, information, and expertise needed to make a quality decision. Consequently, the use of decision support tools to support expertise in the decision-making process is critical for organizational decision-making.

2.1 What is expertise?

Though no agreed upon definition exists within the literature for expertise, researchers would agree expertise is a multidimensional construct [9], with expert knowledge as the

essential part. Three main components make up expert knowledge: (1) formal knowledge, (2) practical knowledge, and (3) self-regulative knowledge [10]. Formal knowledge is explicit where learning is the focus of factual information. For instance, a lawyer would know the laws and case histories from schooling. Practical knowledge develops in the skill of “knowing-how” and is the tacit knowledge, where intuition plays a role making expert knowledge difficult to explicitly express. Lawyers have practical knowledge through their extensive experiences from being in a legal setting which better prepares them to make a legal argument or judgment. The third component, self-regulative knowledge consists of the reflective skills that individuals use to evaluate their own actions. For self-regulative knowledge, a lawyer would monitor his argument, presentation, and reasoning while presenting to the judge or jury. As elusive as a definition is for expertise, its short supply and difficulty to represent makes who possesses it extremely valuable because of its influence on decision-making. Nevertheless, expertise is thought of as a highly specialized or domain-specific [11] set of skills that have been honed through practice for a specific purpose [12] and perform consistently more accurate in relation to others [13].

Figure 1 Conceptual Map

2.2 Capturing Expertise by means of

Technology

Considering expertise is not only restricted to human beings- rather technology’s capacity to posses “expert” ability to influence decision-making, organizations have allocated significant resources to leverage expertise using technology. Each technology or system has been built to better capture knowledge or represent expertise in the cognitive process of the decision-maker(s) for effective decision-making to occur [14][15][16]. Expertise captured and managed from the support systems embody why each system is important for organizations to have. A review of the literature shows that organizations have used different support technologies to support their expertise needs, however opportunities exist for further synergistic integration that has the capacity to support a broader stroke of problems using emerging technologies.

Figure 2 Level of Expertise in System Design

3 Leveraging Expertise in Decision

Support Technology

3.1 Expertise in Expert Systems

One method used by organizations to capture expertise is by employing expert systems. Currently, expert systems are playing a critical role for many organizations and are a source of competitive advantage [17]. Expert systems, a branch of artificial intelligence, are contributing to decision-making through their representation of knowledge and reasoning of human experts for its users [18]. By mimicking and replicating the cognitive process of a human expert, novice users can be supported to perform as well as experts [19] while expert users can have their expertise further refined. By emulating an expert’s problem-solving ability, knowledge and reasoning are transferred to a user through the use of expert systems for faster learning and decision-making than would occur when developing these skills over time. The main function of an expert system is to represent expertise to its users for decision-making when a human expert can not be found or is in short supply. Legitimated as an alternative to human experts in the 1970’s, the expert system MYCIN was found to perform better or equally as well at diagnosing meningitis in blood as human experts [20][21]. Expert systems are constructed with four main components- knowledge base component, heuristic engine component, user interface, and explanation module. The knowledge-base component consists of the factual knowledge a human expert would have of a specific and narrow domain, while the heuristic knowledge or expertise is based on intuition, experience, and judgment to apply rules efficiently under uncertainty or with incomplete information. The user interface component allows the user to interact with the system but it is the explanation module that queries the user for more information. As the user responds by answering the questions presented through the explanation module, the new information is then incorporated in the decision-making process and finally used to provide a justification for solution, which is a critical factor for system intelligence [22]. The end decision, resulting from the dialogue and collaboration between the user and mimicked human expert is valuable for

Expertise in Design

Number of People

Expert Systems

GDSSs

Collective Intelligence Tools

Expert

Systems

GDSSs

CI Tools

Integrated

System

organization. For example, research has [23] found that users’ interaction with the expert system while verifying information may impact whether the user accepts the system’s output for consideration in decision-making. The understanding of how a recommendation or solutions is reached by an expert systems, is possible through a knowledge acquisition process, which shows the line of reasoning generally from one specialized expert why a decision is and should be made. This is to emulate the thought process and reasoning of a human expert. Knowledge-acquisition is the diffusion of problem-solving expertise from some knowledge source to a program [24]. Researchers have highlighted [25], the power from expert systems derives from the knowledge it possesses.

Though many organizations have successfully implemented expert systems to address particular problems in a narrow domain, changing external factors impacting competitiveness and sustainability have forced organizations to approached critical decisions differently. Studies indicate [26], the more complex organizations become the fewer decisions are made by any single individual (or expert system). Rather than rely on expertise from one individual or system for an important decision, organizations turn to groups or teams of experts in the decision-making process. Furthermore, groups of experts may be necessary when diverse subsets of knowledge are required and no single expert has complete knowledge of the problem.

3.2 Expertise in Group Decision Support

Systems (GDSSs)

Interest grew in organizations in the 1980’s to improve decision-making in group meetings as a result of the changing dynamics characterized by greater knowledge, complexity, and turbulence [27][28] that groups face. As organizations turned to communication technology to foster more effective and efficient group decision-making [29] changes in computing power and electronic communication supported new forms of organizational work. Organizational transformation occurred as a result of developments in new decision support systems impacting the organizational structure, project based teams, dispersed workforce, and greater emphasis on collaboration.

One technology supporting organizational change and group decision making is group decision support systems (GDSS). GDSS use has shown to reduce time, costs [30], and foster collaboration, communication, deliberation, and negotiations [31]. Research in group decision support system theory suggests that through communication, collective knowledge, and interaction of participants enables better solutions to be reached over any single individual. When a GDSSs is used in decision-making it aims to improve the process of group decision-making for opinion convergence, group consensus, and better outcomes in decision-making. Designed using the rationale theory of decision-making, GDSSs optimizes the decision-making process by following what is referred to as intelligence, design, and choice [32]. GDSS use enhances decision outcomes by leveraging the

cognitive knowledge of participants by supporting the behavioral and social needs of the group to resolve uncertainty in the group decision making process. GDSSs possess expertise in the cognitive decision-making process using techniques developed within the support system. As a result of this process, GDSSs are able to capture the knowledge and contribution from the individual users collaborating to arrive at a better solution or create a greater sum than the individual parts. In addition to the cognitive expertise, GDSSs occupy the center point for the aggregation of information and expertise from each participant. GDSSs impact on the decision-process outcome depends on the degree of change in communication of the users and when used effectively better outcomes can occur. To continue success in GDSSs use, GDSSs adapted to the market’s organizational and technological needs of the 1990’s by moving primarily to a web-based software allowing for anytime, anyplace meeting, and decision-making. As a result of GDSS’s flexibility in supporting group decision-making, hybrid technologies available within, such as: email, teleconferencing, data analysis tools, heuristics, and AI techniques in modeling large variables forecasts position GDSS’s to play a larger role in organizations. Though GDSSs have supported organizations by utilizing the expertise of the group and providing structure for effective decision-making [33], decision-makers are still constrained by the information they receive to make a decision. Since the quality of group discussion is greatly contingent upon the quality of information brought to the session by the group members, having tools with capabilities to increase available information internally and externally to the organization would be beneficial [34]. In hindsight, what [34] is alluded to is a changing organizational paradigm, away from a half century of support system development and research that centralized decision-making for experts, to a decentralized model of managing external capabilities, resources, and information of the organization. In organizations, decision-makers do not have access to all the information they need when making a decision [35] and thus, effective decisions can be compromised. Three potential reasons why critical information is not accessed by decision-makers could be: conventional methods and technologies insulate information flow to only a select group of people, decision-makers do not ask for all the information accessible to them, or those who have it do not share because of political or social reasons. As a response, emerging collective intelligence tools are helping to alleviate these constraints.

3.3 Expertise in Collective Intelligence Tools

The notion of collective intelligence is based on the premise that the collective judgment of a large group is better at predicting and forecasting future events than individual experts or small groups of experts [36][37][38]. Collective intelligence tools that support information aggregation offer an alternative to the constraints of information flow in decision-making, knowledge work, and complexity in forecasting of

uncertain events. Moreover, the primary goal of collective intelligence tools is to facilitate the summative body of knowledge, information, product and resources of its users. Contrasting sharply to traditional decision support tools, collective intelligence tools democratize decision-making by including many people in and outside the organization into the information gathering and decision-making process. Diverging from traditional thought where high levels of expertise are seen as the best source of decision-making, collective intelligence tools have the ability to harness lower levels of expertise for peak solutions in decision-making [39]. Prediction markets, a CI Tool can be defined as markets that are designed for the purpose of collecting and aggregating information that is scattered among the traders (users) who participate through trading. When a user participates by trading, information can be reflected in the market values in order to make predictions about specific future events [40][41]. Instead of independently-derived individual predictions, predictions markets enable a collaborative evaluation process where many participants make small contributions with a granularity effect. Derived from the efficient markets hypothesis, markets are expected to be the best predictor of unknown future events and should be seen as a complement to executives and experts to aid in information flows to make decisions more quickly and accurately. Much like a real market, traders are rewarded monetarily or through visibility with in the organization based on the accuracy of the information they provide by participating. When individuals buy or sell contracts based on the information they have, they will be rewarded by being the first mover to reflect this new information into the market before others. Seeking to push decision-making down the corporate ladder and information up toward the top and to those who needed it, CI Tools incubate the hidden information that is scattered around the organization or network to be discovered that allows non-experts to produce expert like results when collectively mobilize. By including a large number of people such as rank and file workers or the public into the decision-making process, organizations can create opportunities to augment their expertise needs. Organizations that effectively mobilize a diverse group of people and tap a new reservoir for problem-solving, transform individuals with a low level of expertise for a given problem into an additional method for forecasting and decision-making. Companies that choose to use CI Tools leverage resources of knowledge, information, and problem-solving ability far beyond what they could afford to deploy internally. Use of CI Tools by companies has had some success [42]; however, much is still unknown about these tools. Future challenges may include using collective intelligence tools not as a replacement for experts but as an additional tool in decision-making. Traditional roles of experts may change and represent a mindset shift from answer givers to inquiry mediators in effort to harness the knowledge of the masses in decision-making. Today the internet has made it easier and more cost effect for companies to implement such tools to guide information flow, however it is the organizations choice

and ability to effectively manage the collective intelligence of its resources. Thus a discussion on how organizations can enhance decision-making using the existing support technologies they have and the potential for added benefits from AI methodologies on CI Tools is needed.

Figure 3 Evolving Decision Support Technology

Adapted from [34].

4 Enhancing Decision-Making and

Collective Intelligence Tools

Past attempts to enhance the quality and efficiency of collaborative work through system integration have shown compatibility among ES and GDSS [34]. Research to date however has not included emerging decision support technologies or analyzed the organizational decision support needs based on today’s environment landscape. Given that research in CI Tools is in a formative stage, it is important that potential synergistic interactions with related disciplines and methodologies be conscientiously investigated. Preliminary investigation shows opportunities exist to enhance decision-making and collective intelligence tools as usage matures and influence in demand grows. Foremost, strong synergies exist between the three decision support technologies that have been analyzed in this paper, which when combined have the flexibility to solve a wide spectrum of problems (Figure 3). Each system has a comparative advantage from each other that provides further basis for justification in fusing these three technologies. Further investigation is currently underway to provide a list of system benefits when amalgamating for decision-making.

In addition, techniques from the AI field offer real possibilities to enhance collective intelligence tools use and benefits, much like AI can support current decision support technologies [43]. Design benefits may include: opportunities to simplify the use of CI Tools by transforming these tools from passive agents that collect and aggregate information into active agents that enhance interaction and solicit information from its users based on reasoning and creditability. By adding an intelligent component to CI Tools, opportunities to better manage interaction between the user and technology and the users to other users exist and provide a foundation in making

Unstructured

Problem Structure

Many

ES

Group Size Supported

Few

Structured

GDSS

CIT

DSS

CI Tools easier to use since participation is critical for their benefits to be realized. Enhancements using AI techniques will allow new tools such as prediction markets to be friendlier, smarter, and more sensitive to user behavior and in changes in the users’ environment further propelling growth in usage of emerging technologies. Evolution towards fusing emerging support technologies with long standing decision support systems provide real opportunities to capitalize on synergies for advancement in managing companies expertise within and outside the organization.

5 Conclusions and Future Work

A review of the literature has indicated organizational use of expertise in decision support technologies are changing which reflects the new roles of experts and non-experts in decision-making. This paper has explored issues of design for integration with existing decision support technologies and the benefits AI techniques can play to enhance collective intelligence tools. Each of the research areas described has played an important role for business in the past and present. By bridging each areas literature and in combining their successes, they can continue to support business decision-making in the future. Today it seems inevitable that decision support technologies of the past combined with emerging decision-making tools are ripe for further fusion. Companies who take the necessary steps in design to integrate these technologies that utilize legacy support systems and emerging decision support technologies will be rewarded with a competitive advantage through accuracy in forecasting and problems-solving with more robust support systems. Finally, this paper lays a foundation contributing to the role decision support technologies play supporting expertise in organizations by: (i) showing a perspective of expertise supported decision support technologies that organizations currently use; (ii) structuring how organizational expertise in short supply can be augmented using collective intelligence tools (iii) and emphasizing fusion of existing methodologies, by providing a model for managing internal and external expertise resources together that enhances decision-making (figure 4). Future research is being made to find and analyze real cases and sectors where transversal uses of these systems have been used. In addition, focus will be given to understand how real-time information and learning capabilities can enhance decision-making.

6 Acknowledgments

We would like to thank those within and outside of ESADE who have offered many useful ideas and discussions about the ideas underlying this paper. This research has been partially supported by the AURA research project (TIN2005-08873-C02), funded by the Spanish Ministry of Science and Information Technology and the Commission for Universities

and Research of the Ministry of Innovation, Universities, and Enterprises of the Government of Catalonia.

7 References

[1] A. Ilinitch, R. DÁveni, and A. Lewin. “New Organizational forms and Strategies for Managing in a Hypercompetitive Environment”, Organization Science. Linthicum: May/Jun. Vol. 7., No. 3., 211-221, 1996.

[2] T. Malone. “The Future of Work”, Harvard. Business School Publishing, 2004.

[3] M. Mauboussin. “What Good Are Experts?”, Harvard Business Review. February, 2008.

[4] R. Zammuto, T. Griffith, A. Majchrzak, D. Dougherty, & S. Faraj. “Information technology and the changing fabric of organization”, Organization Science. September/ October, Vol. 18., No. 3., 749-762, 2007.

[5] G. Hamel and B. Breen. “The Future of Management”, Harvard Business School Publishing. Boston, MA, 2008.

[6] J. P. Shim, M. Warkentin, J. F. Courteny, D. J. Power, D. Sharda, and C. Carlsson. “Past, Present, and Future of Decision Support Technology”, Decision Support Systems, Vol. 33., 111-126, 2002.

[7] E. Turban and P. Watkins. “Integrating Expert Systems and Decision Support Systems”, MIS Quarterly, Vol. 10., No 2., Jun. 121-136, 1986.

[8] R. H. Sprague and E. D. Carlson. “Building effective decision support systems”, Englewood Cliffs, N.J., Prentice-Hall, 1982.

[9] R. J. Sternberg. “Cognitive conceptions of expertise”, In P.J. K. M. Feltovich & R. R. Hoffman. Expertise in context. Human and Machine, 149-162. Menlo Park, CA: AAAI Press/ The MIT Press, 1997.

[10] P. Tynjala. “Towards expert knowledge? A comparison between a constructivist and a traditional learning environment in the university”, International Journal of Education Research, Vol. 31., 357-442, 1999.

[11] M. T. H. Chi, R. Glaser, and M. J. Farr. “The Nature of Expertise”, Hillsdale, NJ: Erlbaum, 1988.

[12] P. Jackson. “Introduction to Expert Systems”, Pearson Addison Wesley, 1999.

[13] R. Hartley. “Expert System Methodology: A Conceptual Analysis”, International Journal of Systems Research and Information Science, Vol. 1., 11-22, 1985.

[14] A. Barton. “Experiences in Expert Systems”, The Journal of the Operational Research Society, Vol. 38., No. 10., 965-974. Oct. 1987.

[15] Y. Liou and J. Nunamaker. “Using a Group Decision Support System Environment for Knowledge Acquisition: A Field Study”, Proceedings of the ACM SIGBDP conference on Trends and directions in expert systems, Orlando, Florida, United States, 212 – 236, 1990.

[16] B. Smith. “Collective Intelligence in Computer-Based Collaboration”, Hillsdale, NJ: Lawrence Erlbaum, 1984.

[17] T. Gill. “Early Expert Systems: Where are they now?”, MIS Quarterly, March, 1995.

[18] S. Weiss and C. A. Kulikowski. “A Practical Guide to Designing Expert Systems”, Chapman and Hall Ltd, 1984.

[19] L. Cascante, M. Plaisent, P. Bernard, and L. Maguiraga. “The impact of expert decision support systems on the performance of new employee”, Information Resource Management Journal, Vol. 15., No. 4., 67-78. Oct-Dec. 2002.

[20] E.H. Shortliffe. “Computer-based medical consultation MYCIN”, American Elsevier, 1976.

[21] E. H. Shortliffe and B. Buchanan. “Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project”, 1984.

[22] M. Woodridge and N. R. Jennings. “Intelligent agents: Theory and practice”, The Knowledge Engineering Review, Vol. 10., No. 2., 115-152, 1995.

[23] D. S. Murphy and S.A. Yetmar. “Auditor evidence evaluation: Expert systems as credible sources”, Behavior & Information Technology, Vol. 15., 14-23, 1996.

[24] B.G. Buchanan, D. Barstow, R. Bechtel, J. Bennett, W. Clancey, C. Kulikowski, T. Mitchell, and D.A. Waterman. “Constructing and Expert System, I Building Expert Systems”, F. Hayes-Roth, D.A. Waterman, and D.B. Lenat. 127-167. Addison-Wesley, Reading, MA. 1983.

[25] E.A. Feigenbaum. “The Art of Artificial Intelligence: Themes and Case Studies of Knowledge Engineering”, International Joint Conference on Artificial Intelligence, 1014-1029, 1977.

[26] M. J. Gannon. “Organizational Behavior: A Managerial and Organizational Perspective,” Boston: Little, Brown, 1979.

[27] G. P. Huber. “Group decision support systems as aids in the use of structured group management techniques”, In DDS-82 Conference Proceedings, 96-108, 1982.

[28] G. P. Huber. “Issues in the Design of Group Decision Support Systems”, MIS Quarterly, Vol. 8., No. 3., Sept. 195-204, 1984.

[29] G. DeSantics and R. Gallup. “A Foundation for the Study of Group Decision Support Systems”, Management Science, Vol. 33., No. 5., 589-609, 1987.

[30] B. Gallup. “The impact of task difficulty on the use of a group decision support system”, Ph.D. dissertation, Dept. of Information and Decision Sciences, University of Minnesota, 1985.

[31] D. Kull. “Group decisions: Can a computer help?”, Computer Decisions, Vol. 15., No. 5., 64-70, 1982.

[32] H. Simon. “Administrative Behavior: A Study of Decision-Making Processes in Administrative Organizations”, 4th ed. In 1997, The Free Press, 1947.

[33] S. E. White, J. E. Dittrich and J. R. Lang. “The Effects of Group Decision Making Process and Problem Situation Complexity on Implementation Attempts”, Administration Science Quarterly, Vol. 25., No. 2., 428-440, 1980.

[34] M. Aiken, O. Liu Sheng, and D. Vogel. “Integrating expert systems with group decision support systems”, ACM Transactions on Information Systems (TOIS), Vol. 9., No. 1., 75-95, Jan. 1991.

[35] R. Dye. “The Promise of Prediction Markets: A Roundtable”, McKinsey Quarterly, Number 2, 2008.

[36] R. Hanson. “Decision Markets”, IEEE Intelligent Systems, Vol. 14., 16-19, 1999.

[37] J. Berg, F. Nelson, and T. Rietz. “Accuracy and forecast standard error of prediction markets”, Working Paper, The University of Iowa, 2001.

[38] J. Surowiecki. “The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations”, Little, Brown, 2004.

[39] S. Page. “The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies”, Princeton University Press, 2007.

[40] J. Berg, and T. Rietz. “Prediction Markets as Decision Support Systems”, Information Systems Frontiers, Vol. 5., No. 1., 79-93, 2003.

[41] J. Wolfers and E. Zitzewitz. “Prediction Markets in Theory and Practice”, In S. Durlauf and L. Blume (eds) The Palgrave Dictionary of Economics, McMillan, 2nd ed, 2006.

[42] T. Ho and K. Chen. “New Product Blockbusters: The Magic and Science of Prediction Markets”, California Management Review, Fall, Vol. 50., No. 1., Berkeley, California, 2007.

[43] K. Crowston and T. W. Malone. “Intelligent software agents”, Byte Magazine. 267-272, Dec. 1988.

Differences Between ES, GDSS, CI

Attributes ES GDSS CI TOOLS

Objective Replicate or mimic human experts Facilitate solutions for a group of people

To sum the knowledge and information of many people

Who makes the recommendation (decision)?

The system or heavily weighted if human is involved

The group and/ or systems through ranking The System/ Tool

Major orientation (characteristic) Transfer of expertise (human-machine-human) Build group consensus

Transfer of hard to find information or qualitative to quantitative data

Nature of support Individual or group Group Individual or group

Problem area characteristic Narrow domain Semi/ Unstructured, broad Limited variability

Type of problem treated Repetitive Unique/ not often / important Forecasting/ dispersed collaborators/ Probabilistic

Reasoning capability Yes (deduction) No Yes (depending on the tool (induction)

Assumptions Closed-world Limited to users boundaries Changing

Expertise Level or In-depth knowledge of problem Specific/ Expert Level Dependent on task or problem

All levels including learning capacity with use

Figure 4 Differences between ES, GDSS, CI Tools Adapted from [34]