benchmarking discussion group telluride cognitive neuromorphic engineering workshop 2014
TRANSCRIPT
Benchmarking Discussion Group
Telluride Cognitive Neuromorphic Engineering Workshop
2014
Major outcomes
• We need NE-specific benchmarking to:– Improve the performance of NE systems with apples-to-apples comparison– Convince investors and industry that our systems have high performance
• Benchmarks should be multi-variate, i.e. not just measuring accuracy but also any or all of the following:
– Biological realism – Necessity for tuning – Power consumption or performance per
operation or task– Latency / speed – Noise robustness – Area / technology node / resources
– Learning speed – Adaptability – Robustness in real-world problems – Multi-sensory problems – Usability
• Existing BenchmarksThere are a number of NE datasets already, such as MNISTDVSSpikes.There are a number of good datasets from cognate fields e.g. robot tasks, and spike sorting.
• New Benchmarks– Potential for NE-specific benchmarks encompassing all variables of interest– Potential for benchmarks with predetermine sample statistics, characteristics etc.– Annual (at Telluride) hardware benchmarking with common input (scenes, audio etc.)
• Hosting of Datasets– Giacomo is keen to host on INE website, with mirror sites at e.g. other Universities. UWS
is available to host immediately.– Data will be available under licence – we suggest Open Data Commons Attribution ODC-
BY 1.0.– Datasets should include readme files or ascii headers, and any code necessary to
translate e.g. jAER to MATLAB.
• Dissemination– Giacomo has suggested the possibility of two papers in FNE - a general overview of the
problem, and a specific paper on benchmarking of spatio-temporal pattern recognition systems (both hardware and software). These may form part of a special issue under Michael’s leadership.
– Anyone who wants to participate in these papers, please email Jon Tapson, [email protected]
– Thanks to everyone who participated, esp. Danny who wrote up the meeting notes.
Hard Problems in Neuromorphic Engineering
• We should come up with a list of Neuromorphic Challenges that we think only neuromorphic engineering can solve
• These challenges can raise awareness and drive the field, as the Hilbert Problems did a century ago
• There should be an annual meeting to update and evaluate the latest progress
• This list may be taken as a definition of Neuromorphic Engineering, so we should be careful how we construct it
• Example challenges: – Face recognition within energy and time constraints – Speaker recognition within noise, energy, and time constraints – Adaptive Motor Control – Operant classical conditioning – Limited memory / Limited time response