AUTOMATION IN ACCESSIBILITY

Automated Testing:Web, Mobile, Desktop Sagar BarbhayaTed DrakeCSUN 2016

It shares the progress Intuit has made to incorporate automated testing across all platforms, where there have been problems, and goals for the future. Also shares a journey of transition of manual testing to Automation testing for accessibility.

This presentation is for other companies that are working towards automated testing.

While this presentation provides information on 3rd party testing tools, it is not an endorsement or critique of any product. It reflects our experience as weve researched and implemented testing strategies. Photo: cup of robots by hobvias sudoneighm https://www.flickr.com/photos/striatic/1276095/1

Intuit Accessibility

- Sagar Barbhaya Software Engineer, Accessibility - Ted Drake Principal Engineer, Accessibility

Coop Intern at Intuit with the TurboTax team and partly did a project with Ted Drake who introduced me to this amazing field.Later came back as an intern for the Live Community which is like help support Team and made their product accessible. Joined as a full time Employee in 2015 to drive accessibility automation initiative.

Ted leads Intuits accessibility initiatives via education, training, and working with product managers.The hand represents Intuits Special Needs and Ability Network for employees and their family members

2

AgendaIntroductionHTML Android IOS WindowsResults

HTML componentsWorldSpaceAccessibility library with MartiniTenon/PayPal packagesAndroid componentsLint ArchitectureGoogle accessibility checksIOS componentXcode plugin3

INTRODUCTION30+ productsAccessibility is a top priority.All customers are EQUAL.Manual EvaluationsMulti PlatformEfficiency and Efficacy

Intuit is software company that develops financial and tax preparation software and related services for small business, Accountants and Individuals. One of the main product of Intuit is Turbotax. Since turbotax supports 1000+ plus forms, it is very difficult to evaluate accessibility for each form manually.

Thus decided to automate compliance testing for accessibility.

4

WorldSpace - DequeHosted Testing ServerScheduled scansPublic Facing SitesTimeline of Compliance, not ErrorsSophisticated rules

Worldspace is external server which tests web based products for accessibility compliance. It crawls through webesites and test for all possible web pages.Intuit has used WorldSpace for several years to do automated testing of our production web sites.It could be installed on internal server, which provides access to non-production servers and JIRA integrationWorldSpace tracks compliance. A single error makes a page non-compliant, so timelines are not useful for tracking progress.

http://www.deque.com/products/worldspace/

5

FireEyes Plugin for WorldSpacePlugs into FirebugManual scan of any pageBuild automation scriptsSync with WorldSpace

If you want to test QA server or preprod server, fireeyes comes to rescue.FireEyes plugin allows manual scans and building automation scripts

Takes input as a link to the product and QA server can access worldSpace with the help of this bridge.Basically fireyes acts like a bridge if you want to test QA or preprod server on Worldspace. Fireeyes upload all the results after the scan on Worldspace. So that developers can still go to Worldspace and see all the errors on QA or preprod server

6

QuickBase and Worldspace ScoresCollaborative database for small businessesTesting Tool + Product ImplementationTrack error score and monthly change

QuickBase (http://quickbase.intuit.com/) Easy to use database for building custom projects.

within the organization keeps a tab on the status of all the products and their status on accessibility against latest rulesKeeps a record of thestockholders name, all the scores from Worldspace, the last date of scoring.Worldspace scores is no of critical errors/no of pages scaned.Also we use a system similar to a Credit scorewhere depending on how well the scores are, they are indicated by different colored zones. The colors used are something like in a traffic signal - where green means all is good to go, Yellow indicates that we need to be cautious andwatchful, things might become red soon which is a big STOP or danger that which means we need to stop and revisit the accessibility standards for that particular product.7

Martini Test Library

Intuits Custom UI Test Automation LibrarySelenium based frameworkC.I. / C.D.Low false errors

Martini is an internal tool for UI Test AutomationThis includes a common set of tests that are shared across Intuit teamsThe Accessibility Library was built to focus on most common errors, while triggering minimal false errorsThe initial library was built by the Intuit.com team and has been improved with customer feedbackPhoto: Martini by Tory Barratt Crane https://www.flickr.com/photos/torimbc/4040067721/

8

Martini Test Architecture

Incorporating the accessibility library requires minimum effort. The tests will open a page, parse the dom, and begin searching for elements based on xpaths.The tests are run against these elements, such as checking all images for alt attribute.If there is a violation, errors are generated and reported to a text file for QE integration.9

Acme WidgetsGood pages: 0 errorsBad pages: Lots of errorsOpen Sourcefyvr.net/acme

We needed a stable set of pages to run our tests against. Acme Widgets includes good and bad versions of basic HTML pages. If our tests were solid, the good pages would throw 0 errors or warnings. The bad pages should throw lots of different errors. Access the test site: http://fyvr.net/acme/Download the code: https://github.com/7mary4/a11y-control-test Screenshot test results: tenon.10

Martini Accessibility Library

Customer FeedbackEverything at Intuit includes user testingProduct QE was our customerTurboTax was an early adopterEasy to integratePerformance ImprovementsNeed Mobile Version

Important step was to be able to deliver to a customer and gain feedback.Fuego Web Player Team (CTG) internal team has successfully incorporated this into their automation framework.FEEDBACK Library helped identify issues like alt text missing on the player.Customer empathy sessions were conducted which helped gain deeper insight.

12

Performance IncreaseOne of the things observed was increase in the performanceAutomation helped bump up these numbersTime to run the tests was decreased from 9 minutes to 28.5 seconds.

As customers started using this library, it helped us to get lot of feedback such as when the page structure is complex test takes more time to run. Thus it motivates us to improve performance. Since now there is no manual evaluation, product releases were faster and less pain to QE/Developer and catches any issues before we go live.13

Future GoalsMore sophisticated testsDocumentationDashboard

Since this library is using by all products we should go and see how all products doing in terms of accessibility.14

API-based testing productSophisticated testsEasy to integrate with other toolsRequires passing code to external serverInternal hosting is possible

Tenon.io is a new product that is API based. The team passes code to the Tenon server, which returns test results in JSON. These results can be easily manipulated to work with multiple bug tracking systems, automation tools, and CI/CD tools. http://tenon.ioThe tests are more sophisticated than our Martini libraryTests provide guidance on error importance and solutionsIntuits security team was worried about passing our non-production code to an external server. This can be done securely, but is still a concern. This could be solved by installing Tenon on an internal server, which is something we are still investigating.Pricing is reasonable. Support is great.

15

Nemo-Accessibility

Node-based testing serverOpen source by PayPalChoose your testing libraryCode stays on Internal ServerJSON Response

https://github.com/paypal/nemo-accessibilityNemo is a node.js based automation framework for browser automation.Nemo creates a Node-JS server to run your favorite accessibility automation libraryEasily integrates HTML Code Sniffer, Chrome Accessibility Library, or Deques aXe libraryThis is something we are evaluating as an alternative to Tenon.

16

But Few Test For AccessibilityAccessibility is used for testing

For mobile applications, the accessibility API is used to find elements for automated testing. The majority of documentation about mobile automation is about how to use these elements within the accessibility API to run automated tests.However, there is little information or libraries for checking if the app is actually accessible.

Photo: iphone by Claudio Schwarz https://www.flickr.com/photos/purzlbaum/367889445/https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UIAutomation.htmlhttp://developer.android.com/training/testing/ui-testing/index.html

17

Mobile testing frameworkiOSAndroidOpen Source

http://appium.io/Appium is an open source tool for automating native, mobile web and hybrid applications on IOS and Android platforms. Also Appium is cross platform. It allows you to write tests against multiple platforms such as IOS and Android using same API.18

UI Automation - AndroidAppium at IntuitGoogle Test FrameworksEspressoRobolectricAccessibilityChecks

An equivalent to martini for the web is Appium for Android devices (Google platform).

Appium is an open source automation framework which makes things more easily available for integration.Google suggests test frameworks like Espresso and Robolectric.

Google Accessibility provides checkers like Espresso and Roboelectric in order to test the accessibility amongst Android devices.https://google.github.io/android-testing-support-library/https://google.github.io/android-testing-support-library/docs/espresso/http://robolectric.org/AccessibilityChecks: automated accessibility tests: http://developer.android.com/reference/android/support/test/espresso/contrib/AccessibilityChecks.html

19

INTEGRATING ACCESSIBILITY CHECKS WITH APPIUM2 Key Options AccessibilityChecks logic + Appium AccessibilityChecks as a serviceService SolutionEasier integrationGoogle Updates

Intuit Mobile testing standard frameworkWork to integration google a11y checks with appiumDecided to make it as CI/CD service Service will run behind the scenes!. So teams who will on board to CI/CD will load there project and we will run that service for all mobile apps. Thus we will let them know of possible accessibility errors in their project.

20

Android LintStatic Code AnalysisBuilt Into Android Studio and EclipseEasy to UseProvides solutions and informationPre-build checking

Android Lintis a static code analysis tool built into Eclipse and Android Studio.Each problem detected is reported with a description message and a severity level, thus easy to prioritize.

21

Android Lint Architecture

Reference: http://developer.android.com/tools/debugging/improving-w-lint.html

As you can see here source files for app and .xml needs to parse to the lint tool. And when we build and run the tool we get the output in the form of issue category such as correctness usability, Security, performance, internationalization and accessibility.22

Video on YouTube: https://www.youtube.com/watch?v=jMnWwz6ZFwU23

Accessibility Automation - iOSAppium is standard UI Test FrameworkBuild Standard Tests for AppiumInstruments Apple Framework

Interestingly, Appium is available on iOS too.Would be interesting to see how integration with that would serve both platforms (Android and iOS).https://developer.apple.com/library/tvos/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UIAutomation.html

24

Deques xCode Plugin xCode IntegrationEasy to useDeprecated New tool soon

https://www.youtube.com/watch?v=euEsfNR5Zw4Xcode is the development environment for any iOS product.As a part of the same, they provide a Xcode plugin that helps identify a11y errors.

Is integrated as a part of the build process of the application and like the other warnings/errors in Xcode, we get the ones related to A11y too.

Very less overhead and simple installation by just connecting to the worldspace server through the project.

Deprecated due to fragility with new xcode builds

25

Accessibility Automation - Mac OSXStill in Research PhasePyAtom: Python based UIFrankk: Cucumber

For the OSX platform, development for the Mac, we are still in the very preliminary stages of researching on how this integration could be done.

There are no obvious potential choices for integration into libraries too for this platform.Current automation productsPyAtom is first python based library which fully enables GUI testing for mac applications via the apple accessibility API. It has direct access to the API.Frankk allows you to write structured text tests or requirements using cucumber and have them execute against your Mac OS application

https://github.com/pyatom/pyatomhttp://www.testingwithfrank.com/mac.htmlhttps://developer.apple.com/videos/play/wwdc2015/406/https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/01-introduction.html

26

Accessibility Automation: WindowsNo consistency between productsLargely manual testingAccessibility = Automated testingC++ and .Net

Windows is a challenge considering every product is massively different.Tests are created per product.Strategies around accessibility are simplifying our testing strategies.Windows Desktop product is built using C++.

27

Refactoring LayersQBForms Custom objectsQBForms built on C++Code refactoring in C# and .NetHope to make life easy!!

Since quickbooks desktop is both for mac and Windows there is common layer we use known as QbForms. Now since QBForms was also built using C++ there is very little chance of making it accessible. Thus now effort is to refactor the codebase with C# and .net to make it accessible.28

ResultsLong Tail MonitoringReduced Barrier of EntryGreater AwarenessLower Costs

Long Tail Monitoring: You cant give everything equal attention, especially within a large company. Automated testing provides awareness for the long tail, those products that often get ignored. For instance, we had a product update their wordpress theme and the error rate skyrocketed. We were able to quickly evaluate the product, provide guidance, and they fixed many of the issues the same day. This wouldnt have been discovered as quickly without automated testing. https://en.wikipedia.org/wiki/Long_tail

Implementing accessibility testing has become significantly easier after incorporating this from a centralized resource.

Automated testing forces engineers, QE, and product teams to recognize accessibility errors. This is something they may not have realized prior to the testing. Good testing also provides context and solutions.

Testing at the continuous integration stage greatly reduces the cost for fixing accessibility issues.

29