Innovative Workforce FundFinal Implementation and Reflection Report

Enabler Interactive Pty Ltd

30 July 2018

1. Project details

Date when last updated: 26 July 2018

Version number: 1

Organisation Enabler Interactive Pty Ltd

Project Name Enabler

Contact name and position of person responsible for evaluation

Huy Nguyen, CEO and Co-Founder

Contact details huy@enablerinteractive.com - 0402 207 775

Page 2 of 27

2. Reporting on outcomes

2.1 Service user satisfaction and empowerment

Outcome indicator: Service users have performance data of their workers to decide from.

We expected that Enabler training would result in support workers delivering better quality services, increasing service user satisfaction. We projected that by involving service users heavily in the development of training content, they would be empowered to shape the way services are delivered.

We also intended to build system capabilities that would allow service users to train workers they employed directly, eliminating the need to use service providers and empowering people with disabilities with maximum choice and control. Additionally, we intended to create a database of support workers, which would display their performance results or Enabler ‘star rating’. At the least, the results of workers would be visible to the people who they support, and we considered the more ambitious goal, a publicly visible database of workers that service users could pick and choose from.

We have been unable to achieve this outcome, as we have not yet released Enabler to market. Development of the basic Enabler platform and a quality prototype of training has taken substantially longer than anticipated. Developing Enabler for the user case of a service user who manages their own workforce is still planned, but is not yet complete. Making performance results visible to service users who do not directly employ their workers is unlikely at this stage, due to data privacy issues and the resources required to build this functionality, although it is a concept we are still exploring.

However, we have undertaken other measures in working towards achieving this outcome once we release Enabler to market. We have consistently involved service users in the development and review of Enabler content as intended. We have sought out real stories that people with disabilities have experienced during service delivery and incorporated them into our designs. Our designs and completed modules have been reviewed extensively by members of the community and advocacy groups, resulting in multiple do-overs and ongoing improvements to our modules.

We have collected feedback via:

- focus groups of four or five people with disability for each build of every module (alpha, beta, final polished build). We have found the participants of these groups via personal networks in the community, advocacy groups (for example VicDeaf), and through organisations who find work for people with disabilities such as Epic Assist.

- engaging with prominent members of the community, such as Naz Erdem, who we hope will be Enabler ambassadors

- one on one sessions with service users from our collaborative service provider partners

Page 3 of 27

We also monitor Facebook discussion groups for service users and their families, such as the NDIS Grassroots Discussion group. At this stage, the group helps us to identify common issues that service users are having, which we may be able to address through Enabler. In the long term, we will use groups such as these to monitor wider community satisfaction and engage with service users we might not otherwise be able to connect with. The feedback we have collected to date indicates that:

- Service users believe they would most benefit from Enabler if they were able to create mini scenarios about themselves to use to rapidly train new workers. This was noted as particularly important for service users who engage multiple workers with high turnover, and do not want to spend a long time repeatedly training them to their specific needs and preferences on shift

- Service users believe that many support workers do not currently understand the concept of privacy, boundaries and appropriate conduct, and that training that can teach and assess these things would be very valuable baseline knowledge they want all workers to know

- Service users identify communication as an important skill for support workers, and that Enabler is so far doing a good job of integrating learning this skill into all modules

- The more real stories we include in training the better. We should include more individual character traits and personal preferences not directly related to the delivery of service in our modules. For example, some service users prefer workers not to wear perfume, or don’t enjoy making small talk.

As aforementioned, we are unable to create personalised scenarios for every service user who wants one at this stage. While we would like to achieve this, it presents several logistic and technical challenges, and would take substantial resources. In the interim, while we continue to explore how we could implement this functionality, we have implemented service user input in the following ways:

- Include choices within every scenario that relate to issues such client privacy, respect for personal belongings, boundaries and inappropriate conduct. For example, in some modules the player has free roam around a client’s home. They will be able to do things such as invade the client’s privacy, but if they do so, they will be given a strike. Once specific example of inappropriate behaviour we have included was created from an incident a service user reported to us, where a support worker had taken a box of BBQ Shapes from her pantry and eaten them, without asking or being given any indication that they should help themselves to a client’s food.

- Provide the user with many choices of dialogue when communicating with a client. When respectful and appropriate communication is used, the player will receive a communication point. When the player chooses inappropriate (for example rude or dismissive) communication, they will lose points.

- Include personal preferences of the characters in our scenarios in the Care Plans the player is provided with, such as personal beliefs or preference of activities, and ensure these preferences are relevant to the choices the player makes during assessment. For example, if they are told the client does not like to eat meat, they will be given the choice to give the client a vegetarian meal or a meal that includes meat and to be respectful in their communication about this or not.

Page 4 of 27

We are confident that the inclusion of these principles in our modules will help us to achieve this outcome. Feedback from service users indicates that they agree this will encourage workers to deliver more personalised and quality services.

2.2 Worker skills and engagement

Attract new and suitable workers to the sector

We anticipated that once Enabler rolled out across Australia, it would contribute to growing a skilled workforce because:

- Comprehensive training that can be completed quickly (i.e. Enabler) would allow skilled workers to enter the workforce more quickly than if they had to complete formal qualifications.

- Video-game style training would be more accessible to people from CALD backgrounds because it relies less on literacy and more on actions and visual choices. This would lower barriers for CALD people to entering the support workforce.

- Enabler could improve worker retention because it prepares workers for the realities of the job through practicing realistic scenarios that contain common challenges. At the inception of Enabler, we spoke to a person who had recently completed their Cert IV. They were excited to become a support worker and believed themselves to be prepared, however upon arrival at their first shift they found themselves totally unprepared for the client’s behaviours. They were ill equipped to handle the situation and were put off support work entirely. This person left the industry. A lack of preparedness to handle the challenges of support work is a story we have heard repeatedly.

- Service providers could ask job candidates to complete Enabler scenarios to ensure workers are suitable, helping to increase retention rates. They would be able to assess a worker’s technical skills, communication skills and behavioural tendencies (eg respecting privacy and choice) before hiring them.

We have not been able to achieve this outcome because Enabler is not yet available to the workforce. However, we have undertaken research and experimentation to test our hypothesis and to develop Enabler in a way that will give us the best chance of achieving this outcome:

- We have consulted with industry experts to ensure we cover the most essential topics to allow someone to enter the workforce in our first modules. The results of these consultations have varied, but manual handling, violence and aggression, abuse and neglect and challenging behaviours were commonly recommended as being urgent topics for brand new workers. We have committed to building all of these modules before releasing Enabler as a subscription platform. We will also release these modules for individual sale as they are completed so that the sector can benefit from them while we continue to build content.

- We have conducted user testing during development with people (including but not limited to support workers) from CALD backgrounds. In earlier versions of our modules, the feedback from these sessions indicated that we relied too much on dialogue between characters. Taking this feedback on board, we have replaced dialogue with activities (such as visual representations of information) as much as possible.

Page 5 of 27

- In our latest rounds of testing, we have received positive feedback, with users saying that the language used was mostly easy enough to understand, and that not too much reading was involved. Testers also reported that dialogue choices were helpful for them, as they were informative of what to say in certain situations where they wouldn’t have had the English language skills to know how to communicate these things previously.

- We investigated the use of basic language as another measure to improve accessibility to CALD workers, but this is challenging because we still need to balance this with being able to portray personality and emotion. We plan on translating Enabler into other languages for CALD workers in Australia in the future.

- During testing sessions with support workers, one of the questions we ask workers is how confident they would be in handling a similar situation to the scenario they played in real life. The overwhelming majority answered that they felt prepared or more prepared to deal with similar situations in real life.

Once Enabler has been rolled out as a subscription service, we will survey service providers using Enabler periodically about improvements in worker retention and the usefulness of Enabler in recruitment. We will also survey support workers periodically to measure whether their on the job confidence has improved using Enabler. Improve knowledge retention and skills

We will not been able to measure our achievement of this outcome until Enabler has rolled out and we are able to collect and analyse performance data from several thousand workers over time. However, we have been able to test our hypothesis that scenario based video game training will improve knowledge retention and skills compared to online training on a small scale to date.

We have conducted extensive user testing with support workers ranging from those who have received no training, have received some training but are brand new to the industry, and very senior support workers. During testing sessions, we observed each tester play through Enabler scenarios without interfering and noted our observations of difficulties they faced, how well they performed, how many strikes they received, mistakes that were made, and mistakes that were repeated on subsequent play throughs.

Page 6 of 27

The results of these observations are as follows:

Testers who rated themselves as confident in the subject matter upon completing the training portion of the module

>90%

Testers who completed a scenario with zero strikes on first attempt

5%

Testers who failed scenarios on first attempt

75%

Testers who failed scenarios on second attempt

10%

Testers who made the same or similar mistakes on second attempt

< 1%

Testers who made the same mistake (for example not inspecting all equipment) in a subsequent but different scenario

<1%

One of the recognised benefits of video game style training is the ability to practice a situation repeatedly until the process becomes second nature. Our observations during testing to date support the hypothesis that it is an effective method to improve retention of knowledge. Once we had observed testers, we asked each of them to answer a number of questions:

Do you feel your confidence level before completing this module matched your knowledge on this topic?

The response to this question varied. However, we often found that senior support workers (in years of experience, not age) who had not performed well in the modules identified that they had the knowledge and skills but were not confident in using Enabler, leading to mistakes. Junior support workers and those who had not received much training usually responded that while they had thought their knowledge was good, given that they had received strikes or failed their initial confidence was unwarranted.

In response to this, we began asking some workers to play the module without having completed the tutorial and then asking them to play it after having completed the tutorial. We noticed during our observations that the testers did not appear to struggle with the mechanics once playing the tutorial.

Do you feel confident in your knowledge and skills on this topic after completing the module?

Page 7 of 27

The responses to this question varied greatly, however most testers assessed themselves as at least reasonably confident. We intend to offer more opportunity to practice skills in practice scenarios before proceeding to assessment scenarios in future modules to improve this.

Did you learn anything?

90% of testers responded that they learned at least one thing from all scenarios tested.

How effective did you find this learning experience on a scale of 1 to 10, with 1 being not effective at all and 10 being extremely effective?

The average response to this question has improved over time. Earlier versions of Enabler were rated around the middle of this scale. Primary reasons given were insufficient feedback being provided to the player about what they had done wrong, and gameplay mechanics difficult or counter-intuitive and leading to confusion or distracting the tester from learning or completing tasks.

To address this, we spent substantial time improving gameplay mechanics. For example, we observed testers trying to move towards objects by tapping on them, so we added this functionality. We built tutorials and tested and redeveloped them repeatedly until testers no longer struggled to navigate and interact with the virtual environment.

We added more instantaneous feedback during gameplay, such as emotes that represented how characters felt in response to dialogue choices, and feedback on why strikes were given. Feedback has improved since making these changes, however we are still lacking positive feedback. This is an important feature of video game design principle, but finding the right feedback that is suitable for our serious topics has been a challenge. We are still in the process of designing appropriate instantaneous positive reinforcement features to reward good choices, for example when the player gets a communication point.

Did you find the training portion of the module effective? Did it adequately prepare you for the assessment portion of the module?

For early versions of Enabler modules, a large percentage of testers did not rate training to be effective. The training was not engaging, and testers began clicking through dialogue boxes on the screen without reading them.

To address this, we have reduced dialogue in the training portion, and have tried to include more activities to impart knowledge. However, we are still not happy with the training portion of our modules. The feedback to date indicates that the scenarios played during the assessment portion of the modules are more effective at achieving learning outcomes.

Did you find the assessment portion of the scenario to be effective at teaching learning outcomes in comparison to doing these tasks in real life? If so, how close to a practical experience was it?

Page 8 of 27

During testing of early iterations of Enabler modules, only a few testers identified that our scenarios provided a practical experience. The majority indicated that they did not feel they had actually done the tasks. The main reason identified was because our game design essentially just involved players selecting from dialogue choices. The look and feel of the scenarios was not realistic, and the player avatar did not animate the actions or choices chosen. We did receive good feedback on our inspection mechanic, and have increased our use of this feature.

To make our modules more of a practical experience, we have improved animations, made visual adjustments to the environment with better lighting, artwork and experimented with multiple different camera angles and perspectives. Most importantly, we have altered our designs to give users more ability to freely roam and interact with characters and objects in the environment. We have increased the choices they can make in how they interact with characters and objects also.

Recent feedback indicates that our modules are now more comparable to doing tasks in real life, making them for effective for people who tend to be kinesthetic learners. Most testers now rate Enabler as a very effective learning experience. However, we have encountered a few workers who identify as exclusively kinesthetic learners who still do not find Enabler effective. We will continue to experiment and test gameplay mechanics and designs to make our modules more effective at delivering practical learning.

Five new workers complete three months of work and are positive about their experience with the organisation.

We are not yet in a position to assess this outcome via this indicators, because the Enabler platform has not been released. We will collect data from our partner service providers once we release Enabler.

One thousand existing workers (from organisations) using Enabler platform

As Enabler has not yet been released, there are currently no workers using Enabler as the tool it was designed to be, where they have access to a large library of training at all times.

However, when we release our Manual Handling module to the sector free of charge in August 2018, we will use our networks to promote it heavily to get as many workers using the platform as possible.

Page 9 of 27

2.3 Organisational sustainability

We cannot realistically measure our achievement of this outcome before we have released Enabler as the subscription platform it is intended to be.

The work we have undertaken towards achieving this outcome to date is as follows:

1. Initial testing of the full Enabler platform Version 1 with multiple providers on a small scale.

Organization Number of test workers

Feedback/Results

Hireup 10 Thought the content may not be as relevant to them as the content was built with old service providers in mind.

Thought the system had a lot of potential.

Several of their workers haven’t experienced this type of training before and found it interesting/novel

YellowBridge QLD 5 Thought the system was limited, would like to see care plans integrated and other more complex features like customised environments

They have just rolled onto NDIS as of late 2017 and it has significantly impacted their operations

Australian Unity 2 Our WebGL system didn’t work on their older browsers.

Their workforce uses Windows phones, which do not support Enabler.

CPA 1 Thought it was exciting, had a lot to say about content improvements from an RTO perspective. Wanted to consider a distributor model

CSA 2 Ongoing interest to date.

Interested in more capabilities from the system to help them manage their business.

See the most value in a vast library of mini scenarios about supporting lots of different individuals with varied and unique needs.

Their workforce fluctuates in size frequently, they need flexibility to drop the number of subscribers month to month or Enabler is not financially viable

Page 10 of 27

Organization Number of test workers

Feedback/Results

Headway ABI 1 Content needs a lot of improvement.

They were very interested in exploring the use of the platform in the areas of recruitment with similar functions to psychometric testing

Endeavour Foundation

1 Saw potential but needed to figure out how it would fit into their existing technology platforms. They had excellence insights into the potential of gaming technology as they were developing VR training for people with disability themselves.

They were only interested in the training and did not require any other capabilities from the platform. They did not envision subscribing their entire workforce to the platform, but rather just distributing individual modules to small groups of workers on a recurring basis.

They used the original MVP demo on their community engagement activities

Page 11 of 27

2. We have begun working to address the issues outlined above:

- We are in the process of conducting market research to identify common challenges we may be able to solve with the Enabler platform beyond training.

- we have made substantial changes to our content- we have committed to developing the feature that will allow organizations to share

care plans with support workers through the Enabler system- added the ability to purchase and distribute modules individually for any number

of workers desired for organizations who do not wish to subscribe to all Enabler content

- altered our subscription model to allow organizations to increase and decrease the number of workers subscribed to Enabler on a monthly basis, and to switch licenses between workers. This will also allow service providers to allocate modules to job candidates without incurring extra cost. Managers will be able to do this without removing workers from their list of workers within the Enabler system, making it possible to shuffle licences between occasional workers with ease to cut costs

- consulted with insurers who work in the sector to discuss endorsement of Enabler, which may allow providers to lower their premiums or risk assessments. These discussions have also led to us working with regulatory bodies such as WorkSafe in the design of modules. We will continue to develop these relationships.

- We have made an agreement with one modern service provider to subscribe to Enabler during development over the next 12 months and assist in the design and testing of new features to specifically address pain points and new challenges as they arise. One new feature we will be implementing as a result of this relationship is the ability of service providers to assign non-training tasks and notes to workers that relate to workforce compliance, such as requests to provide a current police check.

3. Continued to seek general feedback from service providers as we develop and improve Enabler. Some encouraging feedback we have received that indicates Enabler is capable of achieving this outcome is that it offers an affordable way to skill up workers without giving them formal qualifications. This was identified as a bonus because NDIS costing means that providers cannot afford to pay highly qualified workers the salaries they demand, but they still require skilled workers to compete and survive in the emerging consumer driven market.

3. Reflections

3.1 Reflecting on implementation

a. Engaging stakeholders: What were the most useful strategies you used to engage and communicate about the project? How/why were they successful? What would you do differently?

The personal contacts of the CEO have been the most effective way to engage stakeholders. Through these contacts, we have been able to build a growing network of support workers for testing, subject matter experts and people with disabilities to inform content development and review modules, and learning and development officers at a range of organisations to make sales and identify the most needed

Page 12 of 27

training. We have also been able to engage with people within government departments and TAFEs to seek alternative ways to sell and distribute Enabler through this network.

We have found that most people find it difficult to envision Enabler through a verbal description, so the most effective strategy to engage potential customers, investors and advocacy bodies has been to meet with people in person and allow them to try Enabler for themselves.

One effective method we have found to engage members of the community beyond focus groups has been by hiring people with disabilities as quality assurance testers. We have worked with Epic Assist, who organised testers on the autism spectrum. These testers were able to provide more insights than other external QA testers, as they were also able to provide meaningful feedback on our content. We also employ a number of people with disabilities in our team, which gives us credibility in the community. Too often, the intended beneficiaries of projects are left out of developments.

However, although engaging with stakeholders has been relatively easy for us, we recognise that many opportunities were wasted because we engaged with potential customers and investors too early. The first iterations of Enabler were, to put it simply, not good. They did not embody the principles of video game design that make this medium engaging and effective. The dialogue and art was unpolished and the content was of poor quality. The user interface and experience were often confusing and not well thought out. The modules were also full of bugs, which were distracting and prevented users from having a good experience. This led to giving a poor first impression and many stakeholders losing interest.

We also undertook quality assurance testing far too early when we still had several known bugs. This meant that finding unknown bugs was unlikely. Feedback sessions on content were also far less effective because the bugs interfered so much with gameplay.

Once it became apparent that our first modules were of such poor quality, we halted customer engagement for a time to work on improving them. However, this was not the best strategy in hindsight, as there were several months where stakeholders were contacting us and we had nothing to show them, losing more opportunities before we halted production to create a demo module.

If we could go back, we would have focused on polishing our game mechanics, design principles, user experience and interface to build a short demo module first. This would have allowed us to thoroughly test these very important aspects of Enabler before investing time and money into building bonafide training modules that needed to be redone several times. We would have then have used this demo module to undertake preliminary large scale customer engagement. We have learned from these experiences and now have a development process that works. Although we will continue to improve game mechanics and technical capabilities of our platform as we are able, we have a formula that has received extensive positive feedback. Our process is now:- Consult with subject matter experts and Cert III and Cert IV to develop learning

outcomes

Page 13 of 27

- Engage with members of the community to find real life examples to create the scenarios that will teach these learning outcomes

- Block out a basic structure of the module and game mechanics/activities that will be included

- Quickly develop this into a rough but playable module without detail or polished dialogue to test new game mechanics, sequencing and to get initial feedback from SMEs and the people on whom the scenarios are based

- Proceed to writing dialogue scripts and creating a playable module that contains all content and is free of known bugs

- Send the module to SMEs and community members for another round of feedback and testing

- Incorporate feedback and build a high-quality, polished version of the module. This might include improving animations, art, dialogue and game mechanics

- Put the polished build through quality assurance testing to find unknown bugs- Final bug fixing - Release

How appropriate was your initial concept? What/did you change or adapt along the way and why?

Our initial concept was good, but our execution was not. We wanted to create scalable, affordable training that was made more effective by using video game technology. This has not changed, but the way in which we originally went about this has changed substantially.

Once we had completed our first two modules in 2017 (Manual Handling and Abuse and Neglect), feedback from service providers, testers and games developers indicated that we had missed the mark:

- We had not utilised video game design principles. In hindsight, the top priority in our small team should have been for a proven, experienced game designer with sound knowledge of these principles. With this skill set lacking from the team, the modules mostly involved clicking through dialogue, which was not engaging and did not provide users with the practice that the medium had the potential to deliver.

- Our content was not cutting edge as we had hoped. We struggled to create training content on our own, but SMEs were unable to provide the ‘fresh approach’ we were looking for. We encountered far more difficulty than anticipated converting traditional training to a scenario based video game format. Game designers did not have the knowledge to ensure learning outcomes were adequately met by game play, and we ran into trouble just basing modules on how real people use support services due to compliance issues. For example, many people with disabilities are transferred using methods that are deemed ‘unsafe’ by WorkSafe, because it is the only financially and logistically viable solution.

After the first round of feedback, we went back and essentially started from scratch. We overhauled our user interface, game mechanics and production process. We implemented fundamental design rules to ensure the modules were engaging, and focused on making the most of the video game format. Instead of using dialogue to teach, we now try to create mini games and activities to deliver learning outcomes. For example, in our new manual handling module, rather than simply have a

Page 14 of 27

character explain the risk of repetitive motion injury, we allow the user to play a little scenario where they need to tidy up a room. Whilst completing their tasks, they are able to see the strain it puts on their avatar.

We have continued to reassess and structure our modules to allow the player maximum practice, recognising that most studies have concluded this is one of the most valuable aspects of video games for training purposes. Our foundation modules consist of a training section and two assessment scenarios where users apply what they have learned to a situation supporting a client. The first assessment is smooth sailing on an ideal shift where nothing goes wrong. The second assessment contains additional challenges.

However, whilst our current formula is a vast improvement, the majority of feedback from support workers in testing indicates that our training section might still not be as engaging and effective as we had hoped. The best way to learn continues to be playing through the scenarios with clients. For this reason, we are considering changing our approach to include other mediums such as reading materials and videos from external resources in the training section, and focusing on building only realistic scenarios. Users would have one or two practice scenarios, then two assessable scenarios that generate official results. This would save us time and allow us to devote resources to producing more personalised scenarios.

We are also still struggling with the decision of whether to focus our training towards making service providers and their workers compliant with regulations or to provide training that provides more useful and applicable skills and knowledge to the realities of the job. When we developed Enabler, we intended the latter, but did not know how much discrepancy there would be between the two.

We anticipated that the NDIS market would push service providers to focus on upskilling their staff to deliver high-quality, person-centred services in accordance with what their clients wanted, but to our dismay, this has not been the case. The majority of feedback we have had from potential customers is that as demand still far outstrips supply, providers are only seeking tick-box training that makes them compliant for the lowest cost possible. We feel that is a business necessity to focus on compliance based training, but find ourselves torn between being able to achieve our social impact goals and being a viable business.

To provide an example, when developing the content for manual handling, the feedback we had from providers was that making our training in line with WorkSafe recommendations would be a big win. These guidelines suggested that equipment should always be used in the manual handling of people. However, when speaking to support workers and clients, it became apparent that many clients do not have equipment, and just building scenarios where clients had equipment and several support workers are unrealistic and not particularly useful to support workers. The person we wanted to base an assessment scenario on is transferred using a particular manual lifting technique as it is the only feasible option for her, something we know to be very common and requiring the worker to have specific skills and knowledge, but ‘too risky’ according to WorkSafe. We therefore did not include it, but need to make a decision on this issue moving forward.

Our concept for our business model was flawed. We based our pricing per user on the average costs of online training, but found after consulting with several potential

Page 15 of 27

customers that we were not affordable. Service providers also did not want to subscribe their whole workforce to Enabler, but rather just have a few licences that they would rotate between their staff to save costs. This approach would defeat the purpose of Enabler, which is designed to give support workers the ability to access relevant modules at any time on the go, such as on their way to a first shift with a client who had needs they had never encountered before. In response, we have reassessed our pricing and subscription style to be vastly cheaper and more flexible. We are also seeking new ways to add more value and functionality to Enabler beyond training. However, our business model still needs some work and will be a primary focus over the next few months.

We recognise that Enabler needs to be available to people with disabilities who self-manage their support workers, but we were overly ambitious with our plan to implement this user case in the short term. The concept of allowing people to create their own mini scenarios to train their workers was also overly ambitious with our available resources. These concepts have been backlogged for the time being.

3.2 Reflecting on impacts and outcomes

a. What are the top two or three things you have learned from doing this project?

1. Pioneering the first video-game format of support worker training is far more challenging and takes far more resources than initially expected

By far one of the greatest challenges we have had to overcome is balancing the ‘fun’ of video games with the seriousness of the training topics. How can we make the training appeal to both young workers who enjoy video games, and older workers who have a more traditional view of training and are not confident using technology? Even creating the tutorial was a challenge, trying to make it informative enough for luddites to use Enabler confidently, without being too frustrating for those who are very tech-oriented.

It has also been very difficult to convert training content into a serious game, as we do not have any examples we can use as a base. We learned after putting our first two modules out for testing that putting the content into the 3D simulation medium was not enough to achieve engaging training. The work we have done to apply and adapt game design principles to create effective training modules has taken substantially more time and resources than we originally anticipated, as without having undertaken such a task before, we did not know how complex it would be. The first lesson we learned regarding this was that studying, adapting and applying game design principles to our platform should have been a preliminary step. Could we go back, we would have focused on developing our game design, mechanics and user interface before attempting to create the content. The content needs to be structured to fit into the game, not the other way around. For example, we now know that to keep a module engaging, we need an action for the player to do every couple of clicks, whether this be interacting with an object, or making a choice about how to move around the environment. This may impact how we structure a module.

Page 16 of 27

We learned that the most engaging and valuable aspect of using a simulated environment to learn, is to allow players to freely roam around and interact with any objects or characters they want. This would also provide more meaningful performance results than when a player is presented with a choice between two actions. For example, they may choose to enter a client’s home and start cleaning before talking to the client. The might snoop through the client’s belongings whilst they are in the shower. However, creating branching outcomes and developing a way to numerically assess actions taken in free roam, such as the order in which tasks are done, will require an enormous amount of resources to build. We are attempting to develop this now, but without a significant input of resources it is unfortunately a long way off.

We set out to make our training scenarios realistic and representative of the ‘real world’, but have learned that to represent this in game play requires multiple branching endings rather than a pass/fail scenario. Real life is not black and white. Throughout a scenario, a worker might make some poor communication choices, but execute technical service delivery without a fault. In that circumstance, the service user might not be thrilled with the service they received, and in real life they might be unhappy with the worker at the end of the shift. However, to create different endings based on every choice the player makes throughout the module would require tracking every action and choice beyond the points system, and creating potentially hundreds of different branching pathways. We feel this is important to achieve our goals with Enabler, but the resources and time investment required are beyond our reach now. We also need to consider whether such improvements are viable if we are to keep Enabler affordable.

Seemingly simple questions such as whether to include sound were more complex decisions than anticipated. Many testers would like sound, but there are issues with using it that are difficult to overcome. Auto text to voice sounds robotic, and this can impact the ability of the dialogue to portray emotion. Using voice actors is very expensive and finding actors that perform well is not an easy task. We then have the issue of how to realistically portray the voice of someone who has speech difficulties without appearing insensitive. In the future, if the resources become available, we would like to employ the people on whom we base our scenarios to record the dialogue to voice.

Another communication challenge we encountered was how, in simulations that rely heavily on a dialogue driven narrative, to represent communication by people who have communication difficulties in text. If a client-character is unable to speak, the player will need to communicate with them in other ways. However, representing these methods where the player can do this via a screen, and in our low poly art style will take substantial experimentation. We are not sure how we will deal with this yet.

Our low poly art style has also created some other challenges. We still believe it was the right style to choose, because it is reasonably cost effective and quick to produce, but because our character models don’t have fingers, for example, it was quite difficult to represent characters communicating using sign language. It took several weeks of consultations with Deaf advocacy groups to find a solution. The low poly style also means our characters can’t have faces, because we would then have the issue of needing to accurately represent facial expressions, which is extremely difficult even in games with very realistic physical representations of

Page 17 of 27

people. Getting this wrong could cause offence or be confusing. After much experimentation, we settled on giving the character models eyebrows to create the illusion of a face, and use emotes that pop up from the character to represent emotions in response to player choices.

A final challenge in converting existing training into scenario-based learning worth mentioning is the very generic nature of training resources available to us. Scenario based training requires actions and choices to have specific outcomes, and there to be ‘right’ and ‘wrong’ choices. However, existing training will provide information such as ‘if you discover your client has been abused, the first step is to support them’. No information is given on how to support them, but in our scenarios, we need to know what to say, what not to say, the limits of what you can do to help them and so on. Finding subject matter experts willing to provide us with these answers has been extremely difficult and caused several delays in designing modules. 2. The challenges facing the sector are more complex than we anticipated.

As a result of the many conversations we have had with people and organizations in the sector, we have begun to reassess the design of our platform as a whole. Our original hypothesis was that one of the key factors in helping organisations survive in the NDIS market would be their ability to deliver quality, person-centred services to attract and retain customers. We believed Enabler would contribute to their sustainability by solving this issue.

However, because demand still far outstrips supply, quality of services has not been a major issue for service providers at this stage to retain customers. The struggles widely reported to us in recent months have revolved around a lack of funding for administration and to pay staff under NDIS pricing.

We also discovered that a lack of quality training was not identified as an issue by the majority of service providers and that many organisations had stopped spending money on training staff. Therefore, it is not enough that Enabler is simply an affordable training solution, it needs to be able to cut costs for service providers in other areas to be an attractive product to most providers.

Page 18 of 27

3. Designing a system capable of assessing worker attitudes, communication skills and attributes such as empathy was too ambitious for the funding period.

We discovered once we started exploring how to design this functionality that there were many complexities we had not considered, including:

- from a technical perspective, how can we design a system that measures the act of not making a choice? Do we need to assign a numerical value to every choice made or not made? Can our system even do this?

- from a practical perspective, if a player is given the choice between three dialogue choices, knowing that they are being assessed, likely to pick a rude or disrespectful response? How can we determine what they would do in real life?

- Is the most appropriate response subjective? - what if workers make several excellent communication skills throughout a

module, but one really awful one? How will be measure their overall performance?

- Are strikes appropriate for poor communication choices when the learning outcomes of the module relate to technical skills?

- If we adopt a system where points are only awarded for good choices, but strikes are not given for poor choices, how do we provide an accurate result for the player?

We are still trying to resolve these complexities. However, in the interim, we have determined a way to measure a worker’s understanding appropriate communication. If they make an excellent communication choice, they will be awarded a communication point (+1 value assigned to the choice) If they make a poor communication point, they will lose a communication point (-1 value assigned to the choice). To avoid having to assign values to every choice made throughout the module, the ratio of total plus points to negative points will be compared at the end. The assessment criteria ‘understands appropriate communication’ will be represented to the player and their employer (where applicable) as ‘excellent’, ‘average’ or ‘poor’.

We still have a lot of research and experimentation to do to build this functionality into Enabler as initially intended.

Have there been any outcomes that have surprised you?We had hoped that gamified training would help motivate workers to complete training on their own time, and expected that in-game rewards would be the best way to achieve this.

We had planned on building a system where points collected via good performance during assessments could be spent on customising the player avatar, such as with new outfits or hairstyles. However, although this appealed to some younger support workers, we were surprised to learn that most workers would prefer real-world rewards such as gaining access to free training.

We did not anticipate the hunger that support workers would have for more training. The majority of workers we have consulted have rated the training they have received as inadequate. They already have a desire to upskill. We will be

Page 19 of 27

exploring how to design real-world rewards and incentives over the coming months.

b. Were there outcomes that you expected to see, but didn’t?

We expected that Enabler training would be the most valuable to modern service providers who have little to no contact with their employees, however the majority of service providers we have engaged with that fit this description are more concerned with worker compliance than the quality of their services in terms of customer service and user satisfaction. We also anticipated that the NDIS would rapidly drive a cultural shift towards a buyer’s market, but this has not yet been the case, because demand still outstrips supply.

c. How did you measure success? How well did this capture project outcomes? Is there other information that would have provided a better understanding of outcomes?

As we have not yet completed the project - the platform is built, but we need to develop a lot more content before we can release Enabler as the training tool it is designed to be - we have not achieved our outcomes and therefore not been able to measure success as a whole.

However, we are attempting to measure our success in progressing towards achieving our outcomes via feedback.

With each round of development (new User Interface, game design, game mechanics, feature implementation), we seek feedback from service provider, new support workers, experienced support workers, other game developers and Quality Assurance testers. We measure success based on the feedback provided.

We consider success to be an increasing percentage of all stakeholders (service users, service providers and investors) providing better feedback with each upgrade made. We aim for a minimum of 75% user satisfaction in each instance. Once we have rolled out, we will be able to collect data and feedback on a large scale to more accurately measure success, such as:

- do overall workforce results improve for workers using Enabler over time?- is Enabler saving its customers money?- have worker retention rates increased?- have there been fewer incidents involving safety issues since Enabler training

was adopted by organizations?- are service users whose workers use Enabler training delivering happy with

their services?

Reflecting on context: what worked, for whom and in what circumstances?

a. What worked well to support the achievement of your intended outcomes? In what situations did it work well?

Outcome: Improve worker skills and engagement

Page 20 of 27

The scenarios based on realistic situations a worker may face while supporting a client have worked well to allow players to learn and practice. They provide support workers with examples of what to do that existing generic training does not.

Providing dialogue choices that range between very rude or inappropriate to ideal works well to prepare workers for these situations, as it provides workers with a potential script to use when finding the right words can be difficult. This was identified to be particularly useful to CALD workers. Workers are also able to learn the potential consequences of saying the wrong thing. In real life, a client may not inform the worker that they are upset or take them to task, but in our training environment, we are able to give the character the opportunity to say what they wished they had said to someone in similar real life situations.

Adding more ‘free roam’ capabilities to our scenarios to make them more representative of real life. In real life, we are rarely prompted to choose between two actions. Free roam in the environment means the player needs to think about what to do, and what order to do things in, making it a very effective assessment tool.

Outcome: Improve service user satisfaction and empowerment

Using real stories to create Enabler scenarios is working well to make service users feel empowered to change attitudes, skills and behaviours of workers to meet their needs.

Outcome: Improve organisational sustainability

Working closely with organisations who don’t have LMS, CRM and admin software they are happy with has allowed us to identify and address pain points for service providers. There are several features that we will be adding to the Enabler platform that we never would have considered building on our own.

Consulting with service providers who already have all of their administrative and management software in place, and wouldn’t use these features of Enabler, has meant that we will be offering different price points depending on the functionality that organizations want.

Page 21 of 27

b. What didn’t work so well in your project? In what situations didn’t it work so well, and why?

Outcome: Worker skills and engagement

- Relying heavily on dialogue between characters to impart learning was not effective. It was not engaging, and was more frustrating than reading because the player had to click to progress the dialogue every few words. This also made Enabler less accessible to people with language and literacy barriers.

- Not including instantaneous positive feedback for good decisions. Rewards are one of the features of video games that makes them engaging, such as receiving coins. We have not yet included this type of feature due to the difficulty in finding something appropriate. For example, if the player received a point for making a good communication choice in response to a client disclosing abuse, a coin popping up would be in poor taste. We are still trying to find a solution for this.

Outcome: Organisational sustainability

As service providers are not yet facing significant pressure to increase the quality of their services to retain clients, the ability of Enabler to train up skilled staff was not enough to make organisations more sustainable. We needed to be able to save them money on training compared to what they were already spending in the NDIS market. This made our pricing model too expensive.

The subscription model that we initially proposed was also not suitable for many organizations, due to the increasing numbers of workers only working a couple of hours a month. Providers could not afford to pay for a worker to have access to Enabler year round when they deliver very few hours of service. We needed to make our subscription model more flexible.

Making Enabler an exclusively subscription based service was also not desirable for many organizations, as they did not need access to all training year round. This was particularly true for smaller organisations who only onboard new staff once or twice a year to care for only a few clients. For this reason, we are making Enabler modules available for purchase individually to make them accessible to these organisations.

c. Knowing what you know now, what would you do differently?

With all the lessons we have learned, we would have approached the development of Enabler in an almost entirely different way. We are now confident in the right direction for future development, but now face the likely reality we will not have the resources to finish building Enabler.

Page 22 of 27

We would have:

- Spent much more time on market research before commencing development of the Enabler hub and put more time into studying the viability of the business model.

- Invested resources in more experienced team members in the early stages of the development. While junior developers are more affordable, this meant that the skills shortages in our team (such as a talented game designer) were noticed later than they should have been.

- Sought feedback from other game developers early on, to perfect our game design before trying to add content. The principles for how content would fit into the training medium should have come first.

- We would have built and polished our demonstration module before tackling a bonafide training module. Developing training content is costly and time consuming, and having to change entire training modules with each gameplay/UI/mechanic improvement took much longer than changing a short demo module.

- Sought feedback from subject matter experts, service providers, advocacy bodies and support workers on the content and game design before commencing development of each module.

- Outsourced content creation as much as possible. It would have saved us time and money in the end to purchase existing content to use, rather than trying to create our own. This would have given us more time and resources to adapt that content to our platform effectively.

- Spent more time perfecting Enabler before showing to potential investors and major customers. Opportunities were lost because we engaged with the community before the product was up to a high standard.

3.3 Reflecting on legacy: how sustainable are the innovations demonstrated?

a. Are you planning to continue with any of the practices or initiatives you tested during the project? Which practices and what will it require to continue with this approach?

We believe that we now have a working formula and viable product, and intend to continue to develop and improve Enabler. We have listened to providers and service users and know that the personalised mini scenarios we have always intended to build have the greatest potential to achieve our desired outcomes. We still plan to release Enabler as envisioned A platform that gives subscribers unlimited access to all Enabler training content 24/7, that can be easily accessed on the go when required.

However, building enough content to release Enabler in this way will require another two years of development. Therefore, out of necessity, we will need to focus on producing foundation modules that introduce workers to topics such as PCAS, abuse and neglect, medication and violence and aggression, and sell these individually as they are completed to fund ongoing development.

We may also need to temporarily pivot our attention away from service providers towards other potential customers, such as TAFEs, who have expressed interest in using Enabler scenarios to supplement course work. This is a matter of needing resources to continue development.

Page 23 of 27

We will continue to collaborate closely with stakeholders as we continue with development. However, in coming months, we will need to prioritise and settle on our target organizations that will inform the development of future functionality of the Enabler platform. The needs of modern service providers such as Hire Up and CSA are quite different from traditional providers. Realistically we will not be able to make Enabler the ideal tool for all service providers.

Our plan to build Enabler for the user case of service users employing their workforce directly needs further consideration. While this would provide people with disabilities greater choice, control and flexibility over their support services by allowing to train anyone to become their support worker, it would also mean that Enabler may enhance the challenge for service providers trying to attract and retain clients. Development of this functionality has been paused while we consider this, and resources will be prioritised for content development first.

b. Please explain how the changes you undertook can be sustained within the NDIS pricing framework?

We have reassessed our pricing to be affordable for service providers within the NDIS base rate per hour of service delivery. We began with the NDIS costing assumption that support workers spend 95% of their time in direct client facing service provision. This leaves 5% of the hourly rate for organisations to pay for administration, staff meetings, training and professional development.

We consulted with a number of our partner organisations to determine what portion of this 5% could viably be spent on training. The average figure given was 50% for total training budget per worker, and 60% of that for online training, which is where Enabler fits in.

We are therefore using the 60% of 5% of the base hourly rate as the first basis for the monthly fee per user.

We are now in the process of collecting the number of hours of services delivered per month from a number of different organisations of varying size to determine our final costing per worker per month. Price per user per month will be scaled down the more workers are subscribed.

In the event that the number of hours delivered by organizations varies greatly per month, Enabler should still be affordable due to the ability to increase or decrease licences on a monthly basis.

c. Is the approach you trialled suitable to be replicated or adapted by other disability service providers? If yes, what types of service providers would be most likely to benefit and what would they need to do?

Enabler will be suitable for most service providers. Some organisations with existing sophisticated e-learning platforms who want to continue to offer training to their workforce from a number of sources may not benefit as much. Integration of Enabler into existing systems will be very challenging if even possible.

Page 24 of 27

However, for most organisations, Enabler offers an efficient way to assign training to workers and monitor their performance. The alpha version of the completed Enabler platform will have the following functionality and user journeys (please note that as development will not be complete until August, screencasts are not yet available. They will be included in the final report).

For service providers who purchase single modules for their workforce (non-subscription)

1. The user journey ideally begins by a provider going to the Enabler website (https://app.enablerinteractive.com) and creating an account as an ‘organization’. However, if the user purchases an Enabler module through our online store (https://www.store.enablerinteractive.com/), they will be prompted to do so using a valid email ID.

a. If they already have an account, they can use this email ID to login to their Enabler account via our website to access the purchased module.

b. If they don’t have an account, they should use that email ID to create an account as an ‘organization’.

2. The provider will then need to login to the Enabler account, again via our website. They will then be taken to the Enabler administrative portal.

3. The provider will have the ability to invite workers to their organization using a name and email address. They can also upload a list of workers at one go using a spreadsheet.

4. Workers will be sent an email asking them to create an Enabler account. Once they have done so, they will be shown with a ‘registered’ status in their employer’s portal.

5. The provider may then select individual workers from this list, or the entire list of workers, and assign training modules purchased by the organization to them. They will only be able to assign the module to as many workers as they have bought licences for. If they need more licences, they can purchase them through the content store within the portal. They will be able to assign free modules (such as our manual handling module) and tutorials to workers without restriction.

6. Workers will then receive an email notifying them that a new module has been assigned to them.

7. The workers can then access the module on their mobile devices using the Enabler app or by logging into their account on the Enabler website.

8. Upon launching the game, the worker will have up to three official attempts to complete the module:

a. At the end of the first two attempts at the assessment, the worker will be provided with 2 choices:i. Re-take Assessmentii. I’m happy with this result

On the third attempt, the result will be logged automatically.

Page 25 of 27

b. The official rating of the worker will be based on the average of their result from each attempt

9. The service provider can view the results of a worker by clicking on their name from a summary list of the workforce in their portal. This will bring up a list of results from all modules the worker has completed.

10. Once the assessment has been completed, the worker will still have access to the module and can continue playing the module as practice. No official results will be logged or reflected back to the provider but the worker will have a list of unofficial activity results displayed only to them through their portal

11. If the provider wants the worker to retake an assessment, the provider will need to either:

a. Use another module license and create a new assignment, orb. Purchase more licenses and then create the new assignments

How workers will access one-off training

The worker will be able to access the training assigned to them from either the web app (WebGL) or mobile app (iOS and Android). The workers will have email alerts when training is assigned to them.

Workers are able to download the Enabler app free of charge, however unless they have been assigned content by their employer, only the tutorials will be available for them to play.

For service providers who want subscribe to Enabler

The automated purchase system for subscription is yet to be implemented. However, we have already allocated the resources for the development of this functionality.

1. Customer selects a subscription plan to purchase from the Enabler website and creates an ‘organization account’

2. Once they log in to the administrative portal, they will be able to:c. Create a list of workersd. Create categories of workers within that liste. Assign subscription licences to the workers they choose (they are able to

maintain a list of workers larger than the number of licences they have purchased)

f. Assign training to workers individually, by category or by entire workforceg. Set due dates for workers to complete the training h. Set up automated recurring assignment of training. For example, they may

want some workers to redo manual handling every six months. i. View worker performance results.

Page 26 of 27

How workers will access training when their employer (service provider) is a subscriber

The worker will have access to all available modules that their employer has subscribed to. They have the option to access the content directly by logging in on the Enabler website, or on their mobile device.

To access content via a mobile device, the worker will need to download the Enabler application from either the iOS or Android stores (free of charge). The worker will need to login to the mobile app using their account credentials.

Workers will be able to play modules as they choose, regardless of whether a module has been assigned to them. If a module has been assigned to them, they will receive an automated email request from their employer to complete it by a set due date.

For individual users who purchase a single Enabler module

From the Enabler public store:

1. The user navigates to store.enablerinteractive.com.2. The user selects a module that is available on the store and confirms purchase3. The user will be taken to a payment panel where they will have options for PayPal

or credit card. 4. The user will need to supply a valid email address for the purchase and create an

Enabler account to access the module.5. The user can play the module by logging into their account on the Enabler

website, or by downloading and logging into the free mobile app. The purchased module will be automatically visible to them upon login.

6. An official record of their results be recorded on their account. They will have three attempts at the assessment. Once the official result has been recorded, they will be able to play the module freely without affecting their result, allowing them to explore the consequences of ‘bad choices’. The worker will retain permanent access to the module, but if they want to update their official record, they will need to re-purchase that module.

From within their account:1. The user logs into their personal account 2. The user visits the store page within their web portal3. The users purchase the module they want from the list of available modules 4. See steps above on how the worker can play the module.

Page 27 of 27