RECOGNIZING PATTERNS IN NOISY DATA USING TRAINABLE FUNCTIONAL STATE MACHINES Faisal Waris OUTLINE BACKGROUND & MOTIVATION - 3 TRENDS Android Auto Cadillac Gesture Pad BMW In-the-air Gestures IDEA WEARABLE DEVICE GESTURE CONTROL OF HMI Twist Tap SwipeLeft Right Escape Twist gesture recognized Escape gesture recognized IMPLEMENTATION Simple Gesture Vocabulary HMI Menu Hierarchy GESTURE RECOGNITION PATTERN RECOGNITION IN NOISY (SEQUENTIAL) DATA Sensors Inherent Noise Fusion: Accelerometer, Gyro, Magnetometer Limb Movement Not precise Variations Vehicle Motion Acceleration Cornering Bumps SEQUENTIAL DATA, PATTERN RECOGNITION TECHNIQUES FINITE STATE MACHINES (FSM) OUTLINE PATTERN RECOGNITION WITH FSM FSM to recognize aaab pattern OUTLINE FUNCTIONAL STATE MACHINES (FNSM*) FSM expressed using functional programming constructs * FnSM is both singular and plural NATURAL TRANSFORMATION TO F# (ML) type M = M of ('s -> M ) let rec start = function | 'a' -> M a1 | _ -> M start and a1 = function | 'a' -> M a2 and a2 = function | 'a' -> M a3 and a3 = function | 'b' -> M ``end`` and ``end`` _ = M ``end`` FnSM for recognizing the aaab pattern STATE-SPACE EXPLOSION How about 3 as and 3 bs in any order? Are 6 states enough? Need exponential # of states to recognize this pattern FNSM FOR 3 AS AND 3 BS IN ANY ORDER let rec fStart = function | 'a' -> fAB 1 0 |> M | 'b' -> fAB 0 1 |> M | _ -> fStart |> M and fAB countA countB = function | 'a' when countA + 1 = 3 -> fB countB |> M | 'a' -> fAB (countA + 1) countB |> M | 'b' when countB + 1 = 3 -> fA countA |> M | 'b' -> fAB countA (countB + 1) |> M | _ -> failwith "invalid input" and fA count = function | 'a' when count + 1 = 3 -> fEnd |> M | 'a' -> fA (count + 1) |> M | _ -> failwith "invalid input" and fB count = function | 'b' when count + 1 = 3 -> fEnd |> M | 'b' -> fB (count + 1) |> M | _ -> failwith "invalid input" and fEnd _ = printf "done"; fEnd |> M 10 FNSM FOR WEARABLE DEVICE SENSOR DATA type SnsrEvent = { Snsr : SensorTypeEnum Ticks : long X : float; Y : float; Z : float} let rec start = function | {Snsr=LinearAcceleration; Z=z} | {Snsr=Gyroscope; Z=z; X=x} | {Snsr=s; Ticks=t} when Matches an event where the sensor is Linear Acceleration and binds the Z-Axis value to the variable z Given an event data structure of this type (record)... We can define FnSM with Pattern Matching code like this F# FNSM MODULE IN PRACTICE - ANDROID function-state recursive type Evaluation function takes a function-state and an event and returns the new function state PLUS whether something was recognized or not OUTLINE HANDLING NOISY DATA let rec start = function | {Snsr=LinearAcceleration; Z=z} when abs z > 5.0 // m/sec2 possible_swing z 1 |> M | _ start |> M and possible_swing prev_z count = function | {Snsr=LinearAcceleration; Z=z} when count < 30 let new_z = updateAvg (prev_z, count, z) possible_swing new_z (count + 1) |> M |{Snsr=LinearAcceleration; Z=z} -> when -> Compute avg. linear acceleration over some count number of events PARAMETERIZE FNSM FOR TRAINABILITY let rec start cfg = function | {Snsr=LinearAcceleration; Z=z} when abs z > cfg.MIN_Z_THRESHOLD possible_swing cfg z 1 | _ start and possible_swing cfg prev_z count = function | {Snsr=LinearAcceleration; Z=z} when count < cfg.COUNT_LIMIT let curr_z = updateAvg (prev_z, count, z) possible_swing cfg curr_z (count + 1) | FnSM parameterized with configuration data Optimal configuration values can be determined through machine learning FNSM OTHER FnSM as partially applied values in other FnSM Hierarchical organization of state machines Call-return semantics / modularization Parallel execution of multiple child state machines E.g. recognize 1 of many gestures at the same time Reactive Extensions Operators Write more compact and/or efficient custom operators for Rx (used with Apache Storm topology) Future: Project Brillo Android for IoT FNSM CONCEPTS INTERIM SUMMARY OUTLINE FNSM TRAINING PARAMETERS z_accel_front_thrsld = f; z_accel_back_thrsld = f; z_accel_avg_front_thrsld = f; z_accel_avg_back_thrsld = f; xy_accel_tolerance = f; avg_over_count = 1.0f; gstr_time_limit = L; xz_rot_tolerance = f; y_rot_tolerance = f; gstr_time_limit = L; x_accel_tolerance = f; y_accel_tolerance = f; ret_x_accel_tolerance = f; ret_y_accel_tolerance = f; xz_rot_tolerance = f; y_rot_tolerance = f; low_z_accel_limit = f Sample configuration parameters FNSM TRAINING PROCESS 2 Labelled training data: Perform each gesture X times and record sensor output Analyze gesture data and define parameterized FnSM for each gesture with initial (best guess) parameter values 1 3 Formulate a suitable optimization problem (fitness function) that uses the FnSM parameter values this is key Run the evolutionary computing optimization algorithm (EA) given the: 1)Training data 2)Parameterized gesture FnSM 3)Optimization formulation To search for parameter values that yield the best value of the objective function 4 FNSM TRAINING OPTIMIZATION FORMULATION p = parm array values from a population individual g G ={Left, Right, Tap, Swipe}, the action gestures fit(p, g) = GC(g) | count(p, g) countNot (p, g) | is the fitness function for a single gesture. This functions value is calculated by running the FnSM parameterized with p over the training set for g. (The Ideal score is 0 when all of the 10 gesture instances are recognized and none any other gestures are recognized). GC(g) = the # of gestures of type g in the training set for g (constant 10 for all gestures). count(p, g) = count of gesture of type g recognized when the p parameterized FnSM is run over the training set for g. (Ideal value is 10). countNot(p,g) = count of all gestures of type [G {g}] recognized when the p parameterized FnSM is run over the training set for g. (Ideal value is zero). countD(p) = total count of any gesture in G recognized in the negative (driving) training set with a p parameterized FnSM. Ideal score is zero. This term is to reduce inadvertent recognition of any gestures. W1 and W2 are weighting factors to control the influence of some of the terms. In summary the fitness function is construed so that a perfect score of 120 is achieved when all 10 instances of a gesture type are recognized with the corresponding training set and none of any other type are recognized, for each of the action gesture types. Step (before / after CA)Fitness Score (max 120) Best guess parameter values -1.5 After CA optimization92.5 SUMMARY Functional programming provides a powerful means of modeling FSM State | Events | Transitions function | pattern match | return a function Partial function application Model complex patterns (state-space compression) Handle probabilistic patterns through aggregation Make FnSM amenable to training or machine learning through parameterization Training: Training data + Optimization + EA Human modelled, machine optimized, computationally efficient pattern recognition Upcoming paper in IEEE Conference Swarm / Human Blended Workshop Human-Defined, Machine-Optimized - Gesture Recognition using a mixed Approach SHBI2015 (Cleveland, Sept. 28, 29) OUTLINE A TAXONOMY OF EVOLUTIONARY COMPUTING ALGORITHMS Evolutionary Computing Biological Genetic Algorithm Genetic Programming Evolutionary Algorithm Neural Net Artificial Immune System Social Ant Colony Optimization Particle Swarm Optimization Artificial Bee Colony Cultural Algorithm Molecular Chemical Reaction Optimization Simulated Annealing Hybrid algorithms + almost endless variations NO FREE LUNCH THEOREM* Essentially states that if a search based algorithm is better on some problems, it will be worse on other problems (as compared to other search based algorithms) Explains the existence of the sheer variety of evolutionary computing algorithms Different search strategies are needed to tackle different problems * Wolpert & Macready (1997) CULTURAL ALGORITHM FRAMEWORK* INSPIRED BY THE PROBLEM SOLVING ABILITIES OF CULTURE Population Space Belief Space accept() influence() Knowledge Sources Historical/Topological/Domain/Normative/Situational *see Wikipedia page for introduction and further references Knowledge Distribution CULTURAL ALGORITHM TYPES F# (ALMOST) type CA = { Population : Population Network : Network KnowlegeDistribution : KnowledgeDistribution BeliefSpace : BeliefSpace AcceptanceFunction : Acceptance InfluenceFunction : Influence UpdateFunction : Update Fitness : Fitness } where: Topology = the network topology type of the population e.g. square, ring, global, etc. Knowledge = Situational | Historical | Normative | Topographical | Domain Individual = {Id:Id; Parms:Parm array; Fitness:float; KS:Knowledge} Id = int Parm = numeric value such as int, float, long, etc. Population = Individual array Network = Population Id Individual array Fitness = Parm array float BeliefSpace = KnowledgeSource Tree Acceptance = BeliefSpace Population Individual array Influence = BeliefSpace Population Population Update = BeliefSpace Individual array BeliefSpace KnowledgeDistribution = Population Network Population KnowledgeSource = { Type : Knowledge Accept : Individual array Individual array * KnowledgeSource Influence : Individual Individual } THANK YOU contact: fwaris / wayne.edu