prototyping*and*user*interface*design*for* augmented ... › content › dam › mathworks ›...
TRANSCRIPT
Prototyping and User Interface Design for Augmented Coaching Systems
with MATLAB and Delphi
Implementa@on of Personal Tennis Coaching System
MATLAB®, MathWorks®, www.mathworks.com Delphi®/Object Pascal, Embarcadero®, www.embarcadero.com
Author: Boris Bačić, PhD [email protected]
Technology-‐supported/Augmented coaching
“Coaching is essen-al to improving as an athlete. Whatever your goal is – a coach will help you see the things you can’t – and help you improve so that you don’t end up to do the same things over again”
(h>ps://youtu.be/9qw7kKSobJg?t=5s , accessed 22 May 2015)
Andy Murray, Professional Tennis Player.
Introduc@on – Development of Augmented Coaching Systems (ACS)
(Bacic, 2004)
Introduc@on Problem descrip.on – Augmented Coaching System (ACS) design and development:
• Scien@fic rigour and valida@on of: – automated elements of qualita@ve analysis of
human mo@on that are inherent to coaching and rehabilita@on
– subjec@ve expert/coach’s assessment of observed mo@on
– machine learning-‐based model to work on previously unseen mo@on paYerns (i.e. tennis swing)
• User Interface (UI) design challenge – How to convey machine learning, capturing expert
insights and other AI concepts, … – applied to mo@on data represen@ng
human spor@ng ac@vity … – to mixed/diverse audience?
(Bacic, 2008)
(Bačić, 2013)
Introduc@on Problem descrip.on – Augmented Coaching System (ACS) design and development:
• Scien@fic rigour and valida@on of: – automated elements of qualita@ve analysis of
human mo@on that are inherent to coaching and rehabilita@on
– subjec@ve expert/coach’s assessment of observed mo@on
– machine learning-‐based model to work on previously unseen mo@on paYerns (i.e. tennis swing)
• User Interface (UI) design challenge – How to convey machine learning, capturing expert
insights and other AI concepts, … – applied to mo@on data represen@ng
human spor@ng ac@vity … – to mixed/diverse audience?
Personal Tennis Coaching System (PTCS)
Applica@on design objec.ves • Providing end-‐users with interac-ve coaching experience, aiming to improve motor learning and stylis@c execu@on of tennis swings
• Assessment and feedback of captured mo@on data to be automated, personalised and flexible
• Integra@ng various research ar@facts – combining qualita-ve analysis of human mo-on and AI approaches designed in MATLAB
Prototyping of ACS/PTCS BACKGROUND – MATLAB prototyping
• Building MATLAB codebase: – proof of concepts, non-‐con@nuous, incremental, cyclic and
inves@ga@ve development; – modular and reusable design; – rapid algorithms prototyping; – visualisa@on; code refactoring; code readability; and – integra@on of heterogeneous code and 3rd party resources.
• User interface and MATLAB codebase prototyping using: GUIDE
• Pladorm-‐independent 3D s"ck figure player ini@ally developed in MATLAB for each frame, redraw s@ck figure topology using line(X, Y, Z, 'Marker', '.')! …
• Other developed applica@ons can be invoked from MATLAB
User Interface (UI) design using Delphi • Mo.va.on for combining Delphi and MATLAB
– 3D Player – need for fast, responsive, accurate and interac@ve animated 3D s@ck figure replay for expert valida@on, visual feedback and mul@-‐pladorm distribu@on with 3D mo@on data set. Standalone ver. or part of UI applica@on.
– Advanced features and rapid UI design • Extending MATLAB development to using Delphi’s visual and run-‐@me components • Principles: Uniform 'look and feel', interac@ve op@ons ('drag and drop', shortcuts, etc.).
– Control of system resources (single CPU instruc@on commands, messaging etc.) and exis@ng Delphi codebase
• Revised Design Objec.ves – Portable 3D Player – Improving interac@ve and visual performance for 3D
animated s@ck figure player without relying on third party graphic libraries – Centralised data management – Client/Server n-‐@er processing integra@ng MATLAB and Delphi
• MATLAB as a computa-onal server • Delphi as a client. Mul@-‐layered applica@on.
– Efficient implementa@on of the concepts including: • Augmented Coaching Systems and underlying AI-‐based architecture • External synchronisa@on for visualisa@on and replay • Orchestra@on of cons@tuent modules
Design and implementa@on op.ons considered
1. "Code Genera@on in MATLAB" – Convenient for rapid prototyping and implementa@on of cri@cal algorithms using C/C++
2. Manually translate and op@mise MATLAB code ✔ … for 3D Player – Time consuming but able to op@mise the target code for various design and
implementa@on objec@ves
3. Design and produce a stand-‐alone applica@on in MATLAB (using GUIDE and callbacks) – Convenient for pladorm-‐independent UI demonstra@ons and
compiled applica@ons distribu@on
4. Design a UI in another programming language according to MATLAB help documenta@on ✔ … for UI design – Combining preferences, programming languages strengths, exis@ng codebase
and RAD programming tools – Delphi integra@on not documented but "MATLAB COM Integra@on" is well
documented as a star@ng point
Solu@on/Results (1/2) • User profiles: Coach, athlete/learner, AI and Kinesiology scien@sts – Coaching view (or Student view) and – Modelling view (or HMMA view)
• UI data import (via start-‐up CLI, Open File dialogue or drag-‐and-‐drop)
• The UI shows: flexible assessment related to common mistakes, coaching rules, coaching drills/scenarios and diverse skill levels
• Assessment of individual swings and summary of the swings collec@on from observed ac@vity
Modelling view
Coaching view
ACS/PTCS: Implementa@on Results
Unpublished IP
User profiles -‐ views
UI and MATLAB COM Server visualisa@ons – Rela@ve and – Absolute assessment (MATLAB plot() func@on)
ACS/PTCS: Implementa@on Results Assessment summary of the swings collec@on
Solu@on/Results (2/2) • UI and MATLAB session control:
– Implemented as "Shopping Cart" concept in eCommerce – Can ini@alise, re-‐start, and shut down MATLAB server – MATLAB server running as a service or an applica@on – Access to MATLAB's: Workspace, Command Window, Help, …
– Two-‐way data synchronisa@on between: UI – MATLAB – n-‐@er data synchronisa@on: 3D Player – UI – MATLAB – mul@-‐layered UI and (animated and interac@ve) 3D Player architectures similar to ISO/OSI model
– Data visualisa@on in MATLAB e.g. plot() graphs invoca@on
– Mul@ple MATLAB commands and scripts invoca@on
MATLAB COM Server – can run as a (hidden) service or as another applica@on (next slide …)
ACS/PTCS: Implementa@on Results
ACS/PTCS: Implementa@on Results
UI and MATLAB COM Server – session control – orchestra@on of cons@tuent models – external synchronisa@on for visualisa@on and replay
MATLAB COM Server – Command Window (next slide…)
MATLAB COM Server – Command Window
UI and MATLAB COM Server – session control – orchestra@on of cons@tuent models – external synchronisa@on for visualisa@on and replay
ACS/PTCS: Implementa@on Results
Discussion Limita.ons • UI prototype constraints:
– Single MATLAB session control – Singleton 3D Player invoca@on
NB: Mul@ple instances of 3D Player could be invoked (with start-‐up file preloading and closing view) from MATLAB server, CLI or File Explorer.
• PTC prototype limita@ons – Pladorm dependent development: COM technology is MS Windows specific – Installa@on and distribu@on issues – Need for local installa@on of MATLAB – Computa@onal power limita@on – need for parallel processing scalability
Advantages • Fast and responsive UI; run-‐@me MATLAB modelling and data analysis. E.g.
there is no need to recompile the UI when: – working directly on MATLAB server (e.g. using Command Window or modifying
data) or – replacing MATLAB codebase modules.
• Rela@ve ease of single machine development and tes@ng vs. distributed client/server hardware architecture
Conclusion and findings • Combining Delphi and MATLAB COM server is:
– Suitable for research, development and concepts communica@on to diverse audience
– Not recommended for applica@ons distribu@on – Not suitable for commercial development
• Both Delphi and MATLAB development env. and code: – are rela@vely easy to use, maintain, test, debug and modify; – (re)gain familiarity for further advancements and reusability – are complemen@ng each others as ‘best of both worlds’ – have available open source codebase established over years
• Delphi/Object Pascal: – is universal sonware development tool, recommended for mul@-‐planorm RAD with similari@es to C/C++
– compared to MATLAB – it lacks rapid R&D capabili@es, equivalent documenta@on and eco system support …
Future work and opportuni@es • Further advancements in ACS for coaching, injury preven@on
and rehabilita@on. • Acquiring and processing mul@modal mo@on data from
diverse input devices using MATLAB’s toolboxes (video, image and signal processing).
• Developing non-‐visual MATLAB components for Delphi (in collabora@on with Embarcadero)
• Extending client/server development and computa@onal power to distributed processing
• Pladorm independent developments • Implement visual development environment for Human
Mo@on Modelling and Analysis (HMMA) to: – bridge kinesiology and AI research – assist with proof of concept tes@ng and – rapid prototyping
Thank you! Ques@ons?
Related work and references 1. Bačić, B. The hypergeometric distribu@on can help reduce cross-‐valida@on incidents: Two case
studies. In: 2014 Mathema-cal Sciences Symposium, p. 8. Auckland, New Zealand (2014) 2. Bačić, B. Connec@onist methods for data analysis and modelling of human mo@on in spor@ng
ac@vi@es. PhD doctoral disserta@on, School of Computer and Mathema@cal Sciences, Auckland University of Technology, Auckland, New Zealand (2013)
3. Bacic, B. Evolving connec@onist systems for adap@ve sports coaching. In: Neural Informa-on Processing -‐ Le>ers and Reviews, vol. 12 pp. 53-‐62. (2008)
4. Bacic, B. A novel generic algorithm for cluster split iB-‐fold cross-‐valida@on. In: 30th Interna-onal Conference on Informa-on Technology Interfaces – ITI 2008, pp. 919-‐924. Cavtat, Croa@a (2008).
5. Bacic, B. Bridging the gap between biomechanics and ar@ficial intelligence. In: XXIV Interna-onal Symposium on Biomechanics in Sports -‐ ISBS 2006, pp. 371-‐374. Salzburg, Austria (2006)
6. Bacic, B. Towards a neuro fuzzy tennis coach: Automated extrac@on of the region of interest (ROI). In: Interna-onal Conference on Fuzzy Systems (FUZZ-‐IEEE) and Interna-onal Joint Conference on Neural Networks (IJCNN), pp. 703-‐708. Budapest, Hungary (2004)
7. Bacic, B. & Zhang, D. H. Evalua@on of ECOS for the discovery of tennis coaching rules. In: 4th Conference on Neuro-‐Compu-ng and Evolving Intelligence -‐ NCEI'04, pp. 93-‐94. Auckland, New Zealand (2004)
8. Bacic, B. Automa@ng systems for interpre@ng biomechanical 3D data using ANN: A case study on tennis. In: 3rd Conference on Neuro-‐Compu-ng and Evolving Intelligence 2003 -‐ NCEI'03, pp. 101-‐102. Auckland, New Zealand (2003)
Extra reading
MATLAB help topics: • Run External Commands, Scripts, and Programs
• Packaging UIs as Apps • MATLAB API for other languages • MATLAB COM Integra@on