cotesys — cognition for technical systems

6
CoTeSys — Cognition for Technical Systems Martin Buss * , Michael Beetz # , Dirk Wollherr * * Institute of Automatic Control Engineering (LSR), Faculty of Electrical Engineering and Information Technology # Intelligent Autonomous Systems, Department of Informatics Technische Universit¨ at M ¨ unchen D-80290 M ¨ unchen, Germany www.cotesys.org Abstract. The COTESYS cluster of excellence a investigates cognition for technical systems such as vehicles, robots, and factories. Cognitive technical systems (CTS) are information processing systems equipped with artificial sensors and actua- tors, integrated and embedded into physical systems, and acting in a physical world. They differ from other technical systems as they perform cognitive control and have cognitive capabilities. Cognitive control orchestrates reflexive and habitual be- havior in accord with longterm intentions. Cognitive capabilities such as perception, reasoning, learning, and planning turn technical systems into systems that “know what they are doing”. The cognitive capabilities will result in systems of higher reliability, flexibility, adaptivity, and better performance. They will be easier to interact and cooperate with. Keywords. Cognition, Robotics, Vehicles, Automation. a CoTeSys is funded by the German Research Council DFG as a research cluster of excellence within the “excellence inititive” from 2006- 2011, see also www.dfg.de/en/research funding/coordinated programmes/excellence initiative. CoTeSys partner institutions are: Technische Universit¨ at M¨ unchen (TUM), Ludwig-Maximilians-Universit¨ at (LMU), Universit¨ at der Bundeswehr (UBM), Deutsches Zentrum f ¨ ur Luft- und Raumfahrt (DLR), and Max-Planck-Institute for Neurobiology (MPI), all in M ¨ unchen. I. Motivation and Basic Approach People deal easily with everyday situations, uncertainties, and changes — abilities, which technical systems currently lack. Unlike artificial systems, humans develop and learn how to extract and incorporate new information from the environ- ment. Animals have survived in our complex world by devel- oping brains and adequate information processing strategies. Brains cannot compete with computers on tasks requiring raw computational power. However, they are extremely well- suited to deal with ill-structured problems that involve a high degree of unpredictability, uncertainty, and fuzziness.They can easily cope with an abundance of complex sensory stim- uli that have to be transformed into appropriate sequences of motor actions 1 . Since brains of humans and non-human primates have suc- cessfully developed superior information processing mecha- nisms, COTESYS studies and analyzes cognition in (not nec- essarily human) natural systems and transfers the respective insights into the design and implementation of cognitive con- trol systems for technical systems. To this end, cognitive scientists investigate the neurobiolog- ical and neurocognitive foundations of cognition in humans and animals and develop computational models of cognitive capabilities that explain their empirical findings. These com- putational models will then be studied by the COTESYS en- gineers and computer scientists with respect to their applica- bility to artificial cognitive systems and empirically evaluated in the context of the COTESYS demonstrators, including hu- manoid robots, autonomous vehicles, and cognitive factories. COTESYS structures interdisciplinary research on cognition in three closely intertwined research threads, which perform fundamental research and empirically study and implement 1 References are too numerous to be included in this paper; a follow-up survey paper will include references and related projects in detail. Fig.1: COTESYS research strategy: Three research disciplines (cognitive and life sciences, information processing and math- ematical sciences, and engineering sciences) work synergeti- cally together to explore cognition for technical systems. Re- search is structured into three groups of research areas: cog- nitive foundations, cognitive mechanisms, and demonstration scenarios. Cognitive mechanisms to be realized include per- ception, reasoning and learning, action selection and planning, and joint human/robot action. cognitive models in the context of the demonstration testbeds, see Figure 1: 1. Systemic Neuroscience, Cognitive Science, and Neu- rocognitive Psychology — Develop computational models of cognitive control, perception, and motor action based on experimental studies at the behavioral and brain level. 2. Information processing technology — Investigate and develop algorithms and software systems to realize cogni-

Upload: others

Post on 12-Feb-2022

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: CoTeSys — Cognition for Technical Systems

CoTeSys — Cognition for Technical Systems

Martin Buss∗, Michael Beetz#, Dirk Wollherr∗∗ Institute of Automatic Control Engineering (LSR), Faculty of Electrical Engineering and Information Technology

# Intelligent Autonomous Systems, Department of InformaticsTechnische Universitat Munchen

D-80290 Munchen, Germany www.cotesys.org

Abstract. The COTESYS cluster of excellencea investigates cognition for technical systems such as vehicles, robots, andfactories. Cognitive technical systems (CTS) are information processing systems equipped with artificial sensors and actua-tors, integrated and embedded into physical systems, and acting in a physical world. They differ from other technical systemsas they perform cognitive control and have cognitive capabilities. Cognitive control orchestrates reflexive and habitual be-havior in accord with longterm intentions. Cognitive capabilities such as perception, reasoning, learning, and planning turntechnical systems into systems that “know what they are doing”. The cognitive capabilities will result in systems of higherreliability, flexibility, adaptivity, and better performance. They will be easier to interact and cooperate with.

Keywords. Cognition, Robotics, Vehicles, Automation.

aCoTeSys is funded by the German Research Council DFG as a research cluster of excellence within the “excellence inititive” from 2006-2011, see also www.dfg.de/en/research funding/coordinated programmes/excellence initiative. CoTeSyspartner institutions are: Technische Universitat Munchen (TUM), Ludwig-Maximilians-Universitat (LMU), Universitat der Bundeswehr(UBM), Deutsches Zentrum fur Luft- und Raumfahrt (DLR), and Max-Planck-Institute for Neurobiology (MPI), all in Munchen.

I. Motivation and Basic Approach

People deal easily with everyday situations, uncertainties,and changes — abilities, which technical systems currentlylack. Unlike artificial systems, humans develop and learn howto extract and incorporate new information from the environ-ment. Animals have survived in our complex world by devel-oping brains and adequate information processing strategies.Brains cannot compete with computers on tasks requiringraw computational power. However, they are extremely well-suited to deal with ill-structured problems that involve a highdegree of unpredictability, uncertainty, and fuzziness.Theycan easily cope with an abundance of complex sensory stim-uli that have to be transformed into appropriate sequences ofmotor actions1.

Since brains of humans and non-human primates have suc-cessfully developed superior information processing mecha-nisms, COTESYS studies and analyzes cognition in (not nec-essarily human) natural systems and transfers the respectiveinsights into the design and implementation of cognitive con-trol systems for technical systems.

To this end, cognitive scientists investigate the neurobiolog-ical and neurocognitive foundations of cognition in humansand animals and develop computational models of cognitivecapabilities that explain their empirical findings. These com-putational models will then be studied by the COTESYS en-gineers and computer scientists with respect to their applica-bility to artificial cognitive systems and empirically evaluatedin the context of the COTESYS demonstrators, including hu-manoid robots, autonomous vehicles, and cognitive factories.

COTESYS structures interdisciplinary research on cognitionin three closely intertwined research threads, which performfundamental research and empirically study and implement

1References are too numerous to be included in this paper; a follow-upsurvey paper will include references and related projects in detail.

Fig.1: COTESYS research strategy: Three research disciplines(cognitive and life sciences, information processing and math-ematical sciences, and engineering sciences) work synergeti-cally together to explore cognition for technical systems. Re-search is structured into three groups of research areas: cog-nitive foundations, cognitive mechanisms, and demonstrationscenarios. Cognitive mechanisms to be realized include per-ception, reasoning and learning, action selection and planning,and joint human/robot action.

cognitive models in the context of the demonstration testbeds,see Figure 1:

1. Systemic Neuroscience, Cognitive Science, and Neu-rocognitive Psychology — Develop computational modelsof cognitive control, perception, and motor action based onexperimental studies at the behavioral and brain level.

2. Information processing technology — Investigate anddevelop algorithms and software systems to realize cogni-

Page 2: CoTeSys — Cognition for Technical Systems

tive capabilities. Particularly relevant are modern methodsfrom Control and Information Theory, Artificial Intelligenceincluding learning, perception, and symbolic reasoning.

3. Engineering technologies — The areas of mechatron-ics, sensing technology, sensor fusion, smart sensor networks,control rules, controllability, stability, model/knowledge rep-resentation, and reasoning are important to implement robustcognitive abilities in technical systems with guaranteed per-formance constraints.

In recent years, these disciplines studying cognitive sys-tems have crossfertilized each other in various ways.2 Re-searchers studying human sensorimotor control have foundconvincing empirical evidence for the use of Bayes estima-tion and cost function enabled control mechanisms in natu-ral movement control. Bayesian networks and the associatedreasoning and learning mechanisms have inspired research incognitive psychology in particular the formation of causal the-ory with young children. Functional MRI images of rat brainshave shown neural activation patterns of place cells similar tomultimodal probability distributions in robot localization us-ing Bayesian filters.

The conclusions that COTESYS draws from these examplesare that (1) successful computational mechanisms in artificialcognitive systems tend to have counterparts with similar func-tionality in natural cognitive systems; and (2) new consoli-dated findings about the structure and functional organizationof perception and motion control in natural cognitive systemsshow us much better ways of organizing and specifying com-putational tasks in artificial cognitive systems.

However, cognition for technical systems is not the mererational reconstruction of natural cognitive systems. Naturalcognitive systems are impressively well adapted to the compu-tational infrastructure and the perception and action capabili-ties of the systems they control. Technical cognitive systemshave computational means, perception and action capabilitieswith very different characteristics.

Learning and motor control for reaching and grasping pro-vide a good case in point.While motor control in natural sys-tems takes up to 100ms to receive motion feedback, highend industrial manipulators execute feedback loops at 1000Hzwith a delay of 0.5ms. In contrast to robot arms, controlsignals for muscles are noisy and muscles take substantialamounts of time to produce the required force. On the otherhand, antagonistic muscle groups support the achievement ofequilibrium states. Thus, where in natural systems predictivemodels of motion are required because of the large delay offeedback signals, robot arms can perform the same kind ofmotions better by using fast feedback loops without resortingto prediction. Given these differences, we cannot expect thatgenerally all information processing mechanisms optimized

2Indeed, Mitchell has pointed out in a recent presidential address at theNational Conference on Artificial Intelligence the next revolution is expectedto be caused by the synergetic cooperation of the computing and the cognitivesciences.

for the perceptual apparatus, the brain, and the limbs of hu-mans or non-human primates will apply, without modification,to the control of CTSs.

II. The Cognitive Perception/Action LoopCOTESYS investigates the cognition in technical systems in

terms of the cognition-based perception-action closed loop.Figure 2(left) depicts the system architecture of a cognitivesystem with multi-sensor perception of the environment, cog-nition (learning, knowledge, action planning), and action inthe environment by actuators. All research within COTESYS isdedicated to real-time performance of this control loop, in thereal world. On the higher cognitive level, the crucial compo-nents comprise environment models, learning and knowledgemanagement, all in real-time and tightly connected to physicalaction. The mid- and long-term research goals in COTESYS

are to significantly increase the functional sophistication forrobust and rich performance of the perception-action loop.

Actuators

Action

Human

Sensors

Perception

Learning &

reasoning

Knowledge &

models

Planning &

Cognitive Control

Cognitive system architecture

Environment /

Production Process

Learning/Reasoning

Knowledge/Models

PerceptionPlanning/Action

Fig. 2: The cognitive system architecture: The perception-action closed loop (left) and the interplay of the cognitive ca-pabilities (right).

The mapping of the technical system operation onto theperception-action cycle depicted in Figure 2(left) might sug-gest that we functionally decompose cognition into moduleswhere one module performs motor action, another one rea-soning, and so on. In order to achieve the needed syner-gies, the coupling of the different cognitive capabilities mustbe much more intense and interconnected as depicted in Fig-ure 2(right). For example, the system can learn to plan andplan to learn. It can learn to plan more reliably and efficientlyand also plan in order to acquire informative experiences to

Page 3: CoTeSys — Cognition for Technical Systems

learn from. Or, perception is integrated into action to per-form tasks that require hand-eye coordination. Futher, per-ception often requires action to obtain information that cannotbe gathered passively. This COTESYS view on the tight cou-pling of the individual cognitive capabilities is important be-cause it implies the requirement of close cooperation betweenCOTESYS’s different research areas.

COTESYS investigates the perception-action loop withina highly interdisciplinary research endeavor starting withdiscipline-specific views of the loop components in order toobtain a common understanding of key concepts, such as per-ception, (motor) action, knowledge and models, learning, rea-soning, and planning.Perception is the acquisition of information about the envi-ronment and the body of an actor. In cognitive science models,part of the information received by the receptors is processedat higher levels in order to produce task-relevant information.This is done by recognizing, classifying, and locating objects,observing relevant events, recognizing the essence of scenesand intentional activities, retrieving context information, andrecognizing and assessing situations. In control theory, per-ception strongly correlates with the concept of observation —the identification of system states that are needed to generatethe right control signals. Artificial intelligence, a subfield ofcomputer science, is primarily concerned with perception andaction; perception is often framed as a probabilistic estima-tion problem and the estimated states are often transformedinto symbolic representations that enable the systems to com-municate and reason about what they perceive.(Motor) Action is the process of generating behavior tochange the world and to achieve some objectives of the actingentity. To produce action, primate brains use a quasi-hierarchyranging from elementary motor elements at lower cortical lev-els to complex “action” sequences and plans at higher levels.Natural cognitive systems use internal forward models to pre-dict the consequences of motor signals to account for delaysin the computation process and filtering out uninformative in-coming sensory information. This cognitive science view canbe contrasted to control theory, where behavior is specified interms of control rules. Control rules for feedback control arederived from accurate mathematical dynamical system mod-els. The design of control rules aims at control systems thatare controllable, stable, and robust and can thereby provablysatisfy given performance requirements. Action theories inartificial intelligence typically abstract from many dynamicalaspects of actions and behavior in order to handle more com-plex tasks. Powerful computational models have been devel-oped to rationally select the best actions (based on decisiontheory criteria), to learn skills and action selection strategiesfrom experience, and to perform action aware control.Knowledge (Models) in cognitive science is conceived toconsist of both declarative and procedural knowledge. Declar-ative knowledge is recognizing and understanding factual in-formation known about objects, ideas, and events in the en-

vironment. It also contains the inter-relationsships betweenobjects, events, and entities in the environment. Procedu-ral knowledge is information regarding how to execute a se-quence of operations. In cognitive science various modelshave been proposed as part of computational models of mo-tor control and learning to explain behavior of human and pri-mate behavior in empirical studies. Most prominent are theforward and backward models of actions for the predictionof the actions’ effects and sensory consequences and for theoptimization of skills. Graphical models have been proposedto explain the acquisition of causal knowledge with youngerchildren. In control systems, various mathematical models,such as differential equations or automata that capture the evo-lution of dynamical systems, are used. Research in artificialintelligence has produced powerful representations for jointprobability distributions and symbolic knowledge representa-tion mechanisms. It has developed the mechanisms to endowCTSs with encyclopedic and common sense knowledge.

Learning is the process of acquiring information, and, respec-tively, the reorganization of information that results in newknowledge. The learned knowledge can relate to skills, atti-tudes, and values and can be acquired through study, expe-rience, or being taught, the cognitive science view. Learningcauses a change of behavior that is persistent, measurable, andspecified. It is a process that depends on experience and leadsto long-term changes in behavior. In control theory, adaptivecontrol investigates control algorithms in which one or moreof the parameters varies in real time, to allow the controllerto remain effective in varying process conditions. Anotherkey learning mechanism is the identification of parameters inmathematical models. In artificial intelligence, a large vari-ety of information processing methods for learning have beendeveloped. These mechanisms include classification learn-ers, such as decision tree learners or support vector machines,function approximators, such as artificial neural networks, se-quence learning algorithms, and reinforcement learners thatdetermine optimal action selection strategies for uncertain sit-uations. The learning algorithms are complemented by moregeneral approaches such as data mining and integrated learn-ing systems (see DARPA Initiative – grand challenges).

Reasoning is a cognitive process by which an individual orsystem may infer a conclusion from an assortment of evi-dence, or from statements of principles. In the cognitive sci-ences reasoning processes are typically studied in the contextof complex problem solving tasks, such as solving studentproblems, using protocol analysis methods (“think aloud”).In the engineering sciences specific reasoning mechanisms forprediction tasks, such as Bayesian filtering, are employed andstudied. Other reasoning tasks are solved in the system de-sign phase by the system engineers, where control rules areproven to be stable. The resulting systems have no need forexecution time reasoning, because of their guaranteed behav-ior envelope. Artificial intelligence has developed a varietyof reasoning mechanisms, including causal, temporal, spatial,

Page 4: CoTeSys — Cognition for Technical Systems

and teleological reasoning, which enables CTSs to solve dy-namically changing, interfering, and more complex tasks.Planning is a process of generating (possibly partial) repre-sentations of future behavior, prior to the use of such plans, toconstrain or control current behavior. It comprises reasoningabout the future in order to generate, revise, or optimize theintended course of action. In the artificial intelligence viewplans are considered to be control programs that can be exe-cuted, be reasoned about, and be manipulated.

III. The Integrated System Approach to CTSsThe demonstrators are of key importance for the COTESYS

cluster. Demonstrators and demonstration scenarios are de-signed to challenge fundamental as well as applied researchin the individual areas. They define the milestones for the in-tegration of cognition into technical systems.

The COTESYS researchers integrate the developed computa-tional mechanisms into complete control systems and embedthem within the demonstrators. The research areas specifythe kinds of experiments they intend to perform in the contextof the demonstrators. They also specify metrics to evaluatethe progress. Thus, the demonstrators become cross area re-search drivers that enforce researchers to collaborate and pro-duce software components that successfully function in in-tegrated cognitive systems. The demonstrators also transferbasic research efforts into applied ones and thereby promotecooperation with the industry.

The focus on demonstrators and integrated system researchis also important as a research paradigm. The cognitive capa-bilities of CTSs enable them to reason about the use of their in-formation processing mechanisms: they can check results, de-bug them, and apply better suited mechanisms if default meth-ods fail. Therefore, their information processing mechanismsdo not need to be hard coded completely. They should still becorrect and complete but through dynamic adaptation ratherthan static coding. This is important because in all but thesimplest cases completeness and correctness come at the costof those problems becoming unsolvable — computationallyintractable at best. For example, computing a scene descrip-tion from a given camera image is an ill-structured problem,checking the validity of statements in a given logical theory isundecidable, computing a plan for achieving a set of goals isintractable for all but the most trivial action representations.

We will explain the interaction between demonstrator re-search and the other research areas using the cognitive fac-tory as an example. The same kinds of interactions betweendemonstration scenarios and the other research areas will berealized by the cognitive vehicle and the cognitive humanoidrobot demonstration scenarios.

The Cognitive Factory – as an Example for the Interactionbetween the Demonstrators and the other Research Areas.The steadily increasing demand for mass customization, de-creasing product life cycles, and global competition require

production systems with an unprecedented level of flexibil-ity, reliability, and efficiency. The equipment of productionsystems with cognitive capabilities is essential to satisfy theserequirements, which must be addressed to strengthen the high-end production in developed economies.

Fig.3: Production chain in the cognitive factory; c©Prof. Zah,iwb, TUM.

COTESYS will investigate a real world production scenarioas its primary demonstration target for cognitive technolo-gies in factory automation. An example production chain in-cludes an industrial robot, autonomously cooperating robots,fixtures, and conveyors to handle and process these parts. Inaddition, it contains an assembly station where human work-ers and robotic manipulators jointly perform complex and dy-namically changing tasks of assembling the parts.

The demonstrator challenges the cognitive capabilities oftechnical systems in important ways. The production chain in-cludes a sheet metal driving machine. Driving – by incremen-tally forming metal sheets through hammering – is a processthat can produce high quality, complex parts with arbitrary3D shapes. With the required flexibility this production stepcan up to now only be performed by human experts. Usingmachine learning and planning methods, the industrial robotwill learn to perform this complex task reliably, flexibly, andaccurately for varying material properties and target shapes.An adaptive action model of the deformation steps allows therobot to optimize and refine its operation with experience. Therobot will even be capable of planning of complex deforma-tion processes including adequate sequences of deformationsteps and the best step parameterization. The cognitive capa-bilities of the robot enable it to economically produce one-of-a-kind automobile body parts in handcraft quality.

Another cognitive aspect of this demonstrator is that it usessensor networks in order to be aware of the operations in indi-vidual machines, robots, and transportation mechanisms. Us-ing sophisticated data processing capabilities, integrated datamining and learning mechanisms, the machines learn to pre-dict the quality of the outcome based on properties of the workpiece and their parameterization. They form action modelsspecific to the situation and use them to optimize productionchain processing.

Another station in the cognitive factory mounts parts intothe car body. The weight of the parts and the complexity of thestep requires joint human robot action. Heavy parts and toolswill be handled by industrial robots and mobile platforms will

Page 5: CoTeSys — Cognition for Technical Systems

provide parts on the fly, such that human workers will be re-lieved from repetitive and strenuous operations and can focuson tasks that require high-level reasoning. The robot learnsinformative predictive models of the workers’ actions by ob-serving them. The predictive models are then used for syn-chronizing the joint actions. To adapt to their co-workers, cog-nitive mechanisms will enable the machines to explain theirbehavior, for example why they have performed two produc-tion steps in a certain order. The machines are equipped withplan management mechanisms that allow them to automati-cally transform abstract advice into modifications of their owncontrol programs.

IV. Research AreasResearch on neurobiological and neurocognitive founda-tions of cognition — Basic research investigates the neu-robiological and neurocognitive foundations of cognition intechnical systems by empirically studying cognitive capabili-ties of humans and animals at the behavioral and brain level.Researchers will investigate, in human subjects, the cogni-tive control of multi-sensory perception-action couplings indynamic, rapidly changing environments following an in-tegrative approach by combining behavioral and cognitive-neuroscience methodologies.

The research task is to establish experimentally how thesecontrol functions are performed in the brain, in order to pro-vide (1) neurocognitive “models” of how these functions maybe implemented in technical systems and (2) guidelines forthe effective design of man-machine interfaces consideringhuman factors. One of the key results for the research ar-eas studying cognitive mechanisms will be a comprehen-sive model of cognitive control combining mathematical andneural-network models with models of symbolic, productionsystems-type information processing. In contrast to exist-ing models that are limited to static, uni-modal (visual) envi-ronments and simple motor actions the COTESYS model willcover cognitive control in dynamic, rapidly changing environ-ments with multi-modal event spaces.Research on perceptual mechanisms designs, implements,and empirically analyzes perceptual mechanisms for cogni-tion in technical systems. It integrates, embeds, and special-izes the mechanisms for their application in the demonstrationscenarios. The challenge for the area is to develop fast, ro-bust and versatile perception systems that allow the COTESYS

demonstrators to operate in unconstrained real-world environ-ments; to endow cognitive technical systems with perceptionsystems that acquire, maintain, and deliver task-relevant infor-mation through multiple sensory modes rather than vast sen-sor data streams. Besides lower level perceptual tasks, theCOTESYS perception modules will be capable of recognizing,classifying, and locating a large number of objects, of con-ceiving and assessing situations, contexts and intentions, andinterpreting intentional activities based on perceptual informa-tion. Perceptual mechanisms at this performance level must

themselves be cognitive. They have to filter out irrelevantdata, focus attention based on an understanding of the contextand the tasks they are to execute. The perceptual capabilitiesinvestigated are not limited to the core perceptual capabili-ties. They also include post-processing reasoning such as theacquisition of environment models and diagnostic reasoningmechanisms that enable CTSs to automatically adapt to newenvironments and to debug and repair themselves.Research on Knowledge and Learning — The ultimate goalof the COTESYS cluster is the realization of technical sys-tems that know what they are doing, which can assess howwell they are doing, and improve themselves based on thisknowledge. To this end, research on knowledge and learningwill design and develop a computational model for knowl-edge processing and learning especially designed to be im-plemented on computing platforms which are embedded intosensor-equipped technical systems acting in physical environ-ments. This model — implemented as a knowledge process-ing and learning infrastructure — will enable technical sys-tems to learn new skills and activities from potentially verylittle experience, in order to optimize and adapt their oper-ations, to explain their activities and accept advice in jointhuman-robot action, to learn meta-knowledge of their own ca-pabilities and behavior, and to respond to new situations in arobust way.

The research topics that define the COTESYS approach toknowledge and learning in CTS include the following: Firstly,the development of a probabilistic framework as a means forcombining first-order representations with probability. Thisframework provides a common foundation for integrating per-ception, learning, reasoning, and action while accommodatinguncertainty. Secondly, a model of “Action Meta-Knowledge”is developed, which considers actions as information process-ing units that automatically learn and maintain various modelsof themselves, along with the behavior they generate. Thesemodels are used for behavior tuning, skill learning, failure re-covery, self-explanation, and diagnosis. Thirdly, a compre-hensive repertoire of sequence learning methods partly basedon theories of optimal learning algorithms. Finally, an embed-ded integrated learning architecture employing multiple anddiverse learning mechanisms capable of generalizing fromvery little experience.Research on action selection and planning — addresses theaction production aspects of cognition in technical systems.These aspects include the realization of motion and manipu-lation skills, the context-specific selection of the appropriateactions, the commitment to courses of activity based on fore-sight, and specific action capabilities enabling competent jointhuman-robot action.

To generate high performance and safe, action planning andcontrol for locomotion, manipulation and full body motion isintegrated. The planning and control system should be capa-ble of working with minimal, non-technical, and qualitativedescriptions of tasks. High performance and safe operation

Page 6: CoTeSys — Cognition for Technical Systems

c©Prof. Ulbrich, TUM c©Prof. Hirzinger, DLR c©Prof. Beetz, TUM

Fig.4: Demonstrator platforms used in the planned scenarios for cognitive humanoid robots. At the left are two humanoid robots(Johnnie and Lola) to be used for walking and full body motion research. Next is the upper body Justin is used for investigatinghighly dexterous manipulation capabilities. Its hand serving a coffee set is shown next to the right. On the right is a mobile robotwith industrial strength arms that serves as the initial platform for the AwareKitchen scenario.

will enable close cooperation with humans. Another focusis to enable cognitive robots to accomplish complex tasks inchanging and partly unknown environments; to manage sev-eral tasks simultaneously, to resolve conflicts between inter-fering tasks, and to act appropriately in unexpected and novelsituations. They even have to reconsider their course of actionin the light of new information. Hence, the long term vision isto develop action control methods and a design methodologyto be embedded into self-organizing cognitive architectures.Research on human factors studies cognitive vehicles,robots, and factories from a human factors and cognitive psy-chological point of view. Particular emphasis is placed onthe interpretation of the environment and the communicationwith humans enabling human-machine collaboration in un-structured environments. The state-of-the-art in all aspects ofhuman-machine communication will be advanced in order toequip cognitive systems with highly sophisticated communi-cation capabilities. To achieve these goals neurobiology andtechnology are to inspire each other and thereby develop thefollowing aspects of cognitive technical systems: advancedinput/output technology, such as speech, gesture, motion, andgaze recognition is created to construct intuitive user inter-faces and dialogue techniques, as well as sophisticated meth-ods to evaluate the multi-modal interaction of humans and sys-tems. The highest and most complex level involves emotion,action, and intention recognition, with which cognitive sys-tems become more human-like. To pursue these goals novelcomputational user models of cognitive architectures and ap-propriate experimental evaluation methods are investigated.

V. Demonstrators and Scenarios

The COTESYS demonstrators provide the other areas withdemonstration platforms and challenges in the form of demon-stration scenarios. The research results from the other re-search areas will be integrated, specialized, embodied, andvalidated in three scenarios:1. Cognitive mobile vehicles: aerial vehicles for explo-ration and mapping, terrestrial offroad vehicles, and collab-orative rescue missions for autonomous aerial-terrestrical ve-hicle teams.

2. Cognitive humanoid robots: the two-legged humanoidrobots JOHNNIE and LOLA are equipped with lightweightarms and multi-fingered hands from DLR. They constitutethe main platforms and their control systems are extended toperform full body motion. The demonstration scenarios willfeature complex everyday activity, complex full body motion,and sophisticated manipulation of objects.3. Cognitive factory: a production line for individualizedmanufacturing of car bodies is considered. Cognitive aspectsinclude skill acquisition, process planning, self-adaptation,and self-modelling. The production line includes autonomousmobile robots with manipulators in order to achieve the nec-essary flexibility of machine usage.

c©Prof. Wunsche, UBM c©Prof. Hirzinger, DLR

Fig.5: Two autonomous vehicles serving as demonstrators inthe COTESYS cluster: The UBM vehicle MuCAR-3 and TheDLR blimp

The AwareKitchen with a cognitive robotic assistant. Oneof the demonstration scenarios for the humanoid robot demon-strators is the AWAREKITCHEN with robotic assistant, wherethe sensor-equipped kitchen is to observe the actions of thepeople in the kitchen, to provide assistance for the activities,and to monitor the safety of the people. For details about thisscenario and results the reader is referred to the companionpaper [1].

REFERENCES[1] M. Beetz, “The AWAREKITCHEN for Cognitive

Robotics,” in Proc. 4th COE Workshop on HumanAdaptive Mechatronics (HAM), (Tokyo, Japan), 2007.