computational intelligence group projects & research...

Computational Intelligence GroupProjects & Research Interests

http://cig.felk.cvut.czDepartment of Computer Science and Engineering

Faculty of Electrical EngineeringCzech Technical University in Prague

EUROSIM 2007

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 2

Team & Scope

● 5 employees

● 6 PhD students

● Datamining, computational intelligence, artificial neural networks, evolutionary algorithms

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 3

Cooperation

● IBM Research Czech Republic

● Sun Microsystems

● Seznam (czech information portal)

● National museum

● 1st and 2nd Medical faculty, Charles University

● Faculty for Human Studies, Charles University

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 4

Upcoming Events

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 5

CIG (Software) Projects

● FAKE GAME – open source data mining tool

● CIV toolkit – advanced algorithms for Cell processor

● MathSC – Mathematica softcomputing toobox

● BlueCar – mobile robot for intelligent rooms

● SiMoNNe – simulator of modular Neural Nets

Being prepared:

● Java OPT – nature inspired optimization package

● PREPit – automated data preprocessing

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 6

FAKE GAME

● FAKE (Fully Automated Knowledge Extraction)

● by GAME (Group of Adaptive Models Evolution)

● Inductive modeling datamining tool

● Implemented in Java, opensourced in 2007 http://sourceforge.net/projects/fakegame/

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 7

CIV Toolkit

● Computational Intelligence and Voice Processing Toolkit on IBM Cell Broadband Engine

● 3x PlayStation3

● HMM, DTW, PSO, SOM, Neural Gas, Genetic Alg.

● http://axon.felk.cvut.cz/civtoolkit

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 8

Data Preprocessing

● Data preprocessing is a corner stone of successful data mining and modelling.

● It involves among others– Data transformation

– Outliers detection and treating

– Missing data imputation

– Data reduction

– Feature selection/Feature ranking

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 9

FAKE GAME Preprocessing Module

● Some of basic preprocessing methods are implemented in the FAKE GAME project.

● The wizard is implemented to guide user through basic preprocessing steps.

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 10

Automated Preprocessing

● Selection and setup of preprocessing methods is very complex.

● To automate selection of preprocessing methods the genetic approach is involved.– Simple Genetic Algorithm

– Linear Genetic Programming

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 11

Introduction to Feature Ranking and Selection

How important is each feature?

Feature Ranking

1. P-length2. P-width3. S-length4. S-width

Reduction

Knowledge

Feature Selection of

dimensionality

Ranks

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 12

Feature Ranking(FR) in FAKE-GAME

● FAKE-GAME tool creates GAME models using Niching Genetic Algorithm(NGA)

● Importance of each feature can be obtained as a side effect of NGA by computing utilization in model building process

● This approach also causes selection of important features by ignoring redundant and irrelevant.

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 13

Example of Feature Selection using FAKE-GAME

● Exapmle with Hypercube data set from UCI data repository

– UNC is a number of unique chromosomes used for feature ranking inside NGA (from 2 to All unique chromosomes)

– First row with bold numbers shows correct rank of features

– Gray background cells are unused features

UNC 1 2 3 4 5 6 7 8 9 102 1 2 3 4 5 6 7 8 9 103 1 2 3 4 5 6 7 8 9 10

1 / 4 1 2 3 4 5 6 7 8 9 101 / 3 1 2 3 4 5 6 7 8 9 101 / 2 1 2 3 4 5 6 7 8 9 102 / 3 1 2 3 4 5 6 7 8 9 10All 1 2 3 4 5 6 7 8 9 10

● Every feature has correct rank

● With growing number of UNC is Feature Selection fewer restrictive

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 14

Parallel Processing of Recurrent Neural Networks

● Recurrent neural networks– Fully connected

– Next state depends on previous state

● Proposed solutions– Reduce number of temporal connection to lower

communication overhead (Brain cortex architecture is similar)

– Introduce the data set parallelism (ensembles of networks)

yt1=d−∑i=1d

x jt−wij

t 2∑k=1

n

ykt mik

t

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 15

Motivation for Use of Multicore Parallel Systems

● Mainstream today

● Powerful, inexpensive hardware for consumer electronics and game consoles

● Effective use of resources? (most programs runs on single core)

● Highly available and with general purpose programming

● Specialized ASICs more powerful then FPGAs

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 16

Suitable Parallel Platforms

● Sony/IBM based (Cell)● 2 AltiVec CPUs, 8 SIMD SPU cores

● Intel based (x86)● 2-8 core CPUs, 80 VLIW cores in the future● SMP, cache coherent, SIMD

● nVidia based (GPU)● High performance computing initiative Tesla● nVidia CUDA C environment● 96-128 cores / chip, NUMA

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 17

Current Work

● Scalable parallel processing of recurrent neural networks (THSOM as the possible representative)

● Usage of multicore CPUs with care to specific architecture constraints– General purpose x86 CPUs (SMP)

– nVidia CUDA capable GPU (NUMA)

– Sony PS3 with Cell CPU

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 18

Algorithms

● GAME – evolution of hybrid inductive models

● THSOM – temporal data clustering

● CEA – continuous evolution of individuals

● DEANN – evolution of neural networks

● ANTCAST – ant colony with castes

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 19

Continual Evolution Algorithm

● Hybrid Genetic Algorithm– Combination of the genetic algorithm and

– gradient-based optimization method

– Variable population size

– Age parametr of the individual

– Sequential replacement of individuals

– Separated encoding of structure and behavior

– Evolution in the continual time dimension,

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 20

Two dimensional evolution in CEA

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 21

Neural Networks Construction

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 22

Evolution Control Process

● Probability functions– Death and Reproduction probabilities

● Balancing Functions

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 23

Evolution Control Process

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 24

TWEANNs

● TWEANN (Topology and Weight Evolving

Artificial Neural Network) algorithms.

● Topology and parameters (weights) are

optimized simultaneously,

– no need to „guess“ the right topology,

– optimal topology is likely to be found.

● Use of Evolutionary Algorithms (EAs).

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 25

(In)direct encodings

● With classic approaches (direct encodings) only relativly small neural networks are possible-> curse of dimensionality

● Indirect encodings allow the compression of information -> small genome encodes large (regular) neural nework.

● Inspiration in nature -> human genome consists of 30 000 genes which encode about 1011 neurons and 1014 synapses!

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 26

DEANN cellular encoding

● Cellular encoding – based on cellular (neural) growth, cell division etc., the program to build a neural network is encoded as a tree.

● Our algorithm DEANN (Developmental Evolution of Artificial Neural Networks)-> the cellular growth is controlled by a biology inspired model of a Gene Regulatory Network.

small tree

encodes large neural network

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 27

Ant Colony Continuous Optimization

● Optimizing parameters of one neuron in GAME

x1, x

2, ... , x

n є R

hybridizing existing ANT methods with gradient search

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 28

Ant Colony Optimization with Castes

Improving ant algorithms using

groups of ants with different

behaviour

● Solving: (A)TSP, SOP, phylogenetic trees

Spaeth, Cooper, Ferguson (2003)

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 29

Applied Computational Intelligence

● Colabroute – datamining from GPS tracks

● Robospace – shape reconstruction from laser scans

● Spiral – parkinson disease recognition

● BlueCar – mobile robot for intelligent rooms

● ParrotTalk – parrot speech analysis

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 30

Colabroute

● Datamining from GPS tracks

● Automated construction of routable road maps

● Automated extraction of points of interrest (Fuel stations, dangerous crossroads, ...)

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 31

Robospace

● Reconstruction from unstructured vector clouds

● Self-organizing Maps

● Application in mobile robotics

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 32

Parkinson Disease Recognition

● Analysis of spirals drawn by hand

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 33

Parrot Speech Analysis

● Speech Recognition Methods applied to analysis of voices of grey parrots.

● Clustering of samples by Self-organizing Maps

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 34

Mining Biological Signals

● Interest in data mining mainly but not limited to medical applications.– Sleep stages recognition based on EEG

– Heart contractions shapes classification based on ECG

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 35

Parallel Subsolutions for TSP

● Solving subproblems in parallel on Cell

● Updating pheromone on main CPU

clustering: k-means http://www.playstation2.cz

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 36

Anthropological data modelling

In this project, we focus on processing Anthropological data by means of several data mining methods. The goal is to predict an age of individuals described by a set of parameters measured on their skeletons. Data in this project are problematic due to very high noise. Methods are tuned and parameterized to give best possible performance on data. The performance of methods is compared and the recommendation, how to process noisy and partially inconsistent data will be one of the final conclusions of this project.

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 37

Estimation of Fetal weight

● Find accurate model of fetal weight prediction ● Based on sonography measured data during

pregnancy shortly before delivery

● EFW = 0,0504AC2*16,427AC + 38,867FL + 284,074