Stabilizing the CI By Re-runing Flaky iOS XCUI Tests

Published by Shashikant Jagtap on

Apple has released Xcode UI Testing a.k.a XCUI Test framework at WWDC 2015 which enable UI testing of iOS application straight from Xcode without any third party tools like Appium, Calabash or KIF.  These tools that call themselves mobile testing frameworks but they’re actually little more than wrappers to UIAutomation or Instruments.  The release of iOS 10 has broken all these open-source mobile test automation frameworks as Apple stopped supporting Instruments technology. The only options remained was to use XCTest framework from Apple or wait for those tools to build a wrapper around XCUITest.  XCTest is not a new framework but it has evolved quite well with Xcode releases. You can read more about Pros and Cons of using XCTest for iOS app testing here

XCUITest + Continuous Integration Problem

XCUITest framework is still new and some users already complained it’s very fragile especially when we run against Continuous Integration (CI) server. XCUI tests seems to be passing locally but those tests are nondeterministic on CI. You can find lot’s of questions about XCUITest on StackOverflow. The result of that is, CI pipeline become unreliable and  untrusted. The teams keeps ignoring  failing UI tests and everybody loose trust on those UI Tests which can become reason for Continuous Integration failures. The broken UI tests are always being ignored by the development team. However,  Continuous Delivery won’t allow broken build to be deployed to production. Now, everyone have attention to broken UI Test. In order to fix the problem team might have done the following to ensure your UI tests are as resilient as possible.

  • Followed ‘Ports and Adapter‘ pattern to cover all business logic at domain layer without writing any UI Tests. The good example of this would be writing faster iOS acceptance tests or contract tests with Fitnesse
  • Considered  testing UI in isolation using stub backends or mock web server to guard against unexpected data to replace real word UI tests.
  • Added XCUI computed retry  to find UI element on application or loop until XCUIElement found.
  • Resetting simulators on each CI build or each test suite.
  • Added Static wait to XCUITests to make tests reliable.

After doing all this efforts, perhaps few tests are still non-deterministic and  fails randomly and people gets frustrated. This is general behaviour of the UI tests, they are fragile, brittle and hard to maintain, no matter how much efforts you put to fix or how much you defend against it peculiarities of the runtime environment can conspire against you.

Now that, your release to production is blocked because of the flaky UI tests. What to do now ?

Solution

In the scenario mentioned above, we can try one sensible thing. Let’s take just failing XCUITests and try to re-run them but hang one, there is another problem. Apple’s command line tool ‘xcodebuild‘ still not have easy way to run single XCTest from command line. It can take Xcode schemes to test but it’s bit tricky to pass individual tests to ‘xcodebuild’.  The new version of ‘xcodebuild’  has ‘only-testing’ option but it’s bit hard to implement with script. Fortunately, there is a fastlane plugin called ‘fastlane-plugin-setup_fragile_tests_for_rescan‘  to rescue from the problem of flaky XCUI Tests. The full credit goes to author of the plugin ‘lyndsey-ferguson‘ . This plugin does following things

  • Takes failed tests from generated Junit reports from fastlane scan
  • Modifies specified XCUITest scheme to skip passed test and failed tests to scheme
  • Re-runs ‘scan’ 3 times to see the results of test.

This make sense to fix the current CI and get going but ideally,  non-determinstic test should either be fixed for good or deleted however sometimes the world is just not that perfect, maybe for example you aren’t able to improve underlying infrastructure issues etc etc. Hence this may still be a valid course of action.

Let’s see that in Action

Just to demo re-runing failed XCUI tests, I have created an demo iOS app called ‘CoinTossing‘. This application has a “Toss” button and the textField showing result of toss either “Heads” or “Tails”. We also have couple of XCUI Tests to check the results. The probability of failing test is 50%.

Create XCUITest Scheme and Share it

Let’s create new scheme for XCUITest and call it “CoinTossingUITests” and make it ‘Shared” so that CI can see it and  we can pass to fastlane scan.

UITheme

 

Setup Fastlane with Plugin

Now that, we will setup fastlane from Gemfile and bundler. We need to have ‘bundler’ gem installed.

Add fastlane to ‘Gemfile’

Now we can install bundle with command

$ bundle install

This will download all the required fastlane tools. We will create ‘Fastlane’ directory manually and add empty Fastfile.

Now we need to add the fastlane plugin to the project using command.

This will ask permission to change ‘Gemfile’ type ‘y’ to allow. This will create another file in the ‘Pluginfile’ inside ‘Fastlane’ directory.

Now we have setup our plugin which is ready to use. Let’s add the following code to Fastfile.

Note that, we are passing ‘CoinTossingUITests’ scheme to the fastlane scan and  running scan 3 times if test failed. Now that, we should able to run ‘test’ lane

We should be able to see the tests running and if test failed only failed tests will be run again. This will modify out original ‘CoinTossingUITests’ scheme.

Re-run

 

Setup CI Server with Travis

The Fastlane plugin  ‘fastlane-plugin-setup_fragile_tests_for_rescan‘ modifies the scheme and add only failed tests for re-running. It uses ‘xcodeproj’ gem to modify the scheme settings which is the same library that Cocoapods use to change Xcode settings and files. This plugin is  not designed to run locally. The idea is to run them on CI server where we don’t care if scheme is getting modified as we checkout fresh source code for every build. Let’s add TravisCI config to the project so that we can run it on CI server.  Now, enable repository to run on TravisCI and Create ‘.travis.yml’ file with following code

Now, we should be able to run those tests on TravisCI. Checkout the latest output from TravisCI here

TravisCI

Try Yourself

The source code for this demo is available on GitHub repo ‘CoinTossing‘. In order to quickly check how plugin works. Clone the project and try running test. Assuming you have Ruby environment setup,  you can run following commands to test this

Now, you can see the tests starts running in the simulator. Observe the test output, if one the test fails it should see that plugin will re-run that tests couple more times as shown in the above GIF.

Things to Remember

Now that, we should be ale to re-run failed XCUI tests in order to make CI happy and carry on however there are couple of things we should remember before we do this

  •  Do not run tests with Fastlane plugin locally. This plugin modifies the scheme and we have to make sure we revert the changes made by plugin. This plug should only be used on CI server
  • The flaky and non-determinstic test should either be fixed for good or deleted
  • Keep the retry ‘scan’ count 3 if test fail after couple of time then there is real issue with the test.
  • Currently we are clearing simulators if retry count is more than 1 but it’s up to you if really wanna clear the simulators.

Conclusion

Apple’s new Xcode UI Testing framework might be little flaky on CI server but we can re-run the flaky tests using the Fastlane plugin to make the XCUITests more robust and reliable. Hopefully this will help to reduce your pain  of flaky XCUI Tests on CI server.