The Fast, Furious and Flaky Continuous Integration with Xcode 10 Parallel Tests

Published by Shashikant Jagtap on October 22, 2018

Apple is truly investing in testing tools and technologies which is a great thing for app developers. There is no doubt that everyone appreciates their work to motivate them to keep doing it.  Apple announced parallel testing support for XCTest framework at the WWDC 2018, platform state of union session. There was a separate session on What’s New in Testing describing new features in Code Coverage, XCTest, and XCUITests including the parallel testing. They actually announced the parallel testing feature last year WWDC 2017, where we can specify multiple destinations to trigger on different simulators or devices accordingly. However, with Xcode 10 we can split the tests in the different clones of the same simulator. Xcode creates a different runner process under the hood and each process gets specific tests assigned. This supposed to reduces the test execution time dramatically. In this post, we will see what’s good and what areas need improvements.

Parallel Testing in Xcode 10

With Xcode 10, parallel testing can be easily enabled by editing the UI Test scheme. We can select the “Test” option, select “Options” against the test bundle to choose parallelization option.

By default, there is only two runner process assigned but we can increase the number when running from the command line. The various option to run XCTest in parallel has been added to the  xcodebuild tool like -parallel-testing-enabled YES|NO  and  -maximum-parallel-testing-workers NUMBER  and many more. These options allow us to execute the XCTests in the parallel.

Does Parallel Testing Works Locally?

With Xcode 10, we can try this feature on the pilot project. Let’s create an iOS app which has the “Enter” button on the home screen. In our UI test. we will launch an app and assert that the ‘Enter” button is there. Pretty simple UI test, right? It shouldn’t fail or shouldn’t be flaky. I have created a demo app available on Xcode10-ParallelTest-CI Github. In order to run the test locally, we can edit the scheme and enable Parallel test from the test action as shown above. You can get the demo source code and try it yourself by downloading this repository from Github. Once pressed CMD+U from Xcode, the tests should launch two clones of the simulator and run in parallel. There won’t be any issue running them locally.

Yes, parallel testing works well on the local machine!

Does Parallel Testing Works on CI?

Now that, we have confirmed that parallel testing feature works on the local machine. Let’s configure parallel tests on Continuous Integration server using Travis CI. In order to do this, create a .travis.yml file at the root of the project with below code

This will create a build matrix for iPhone X and iPhone 6 simulator to run our tests. When the build is triggered on Travis CI, we can see that all tests passed in just 3 minutes.

You can check out the entire build logs on Travis here

This confirmed that parallel testing works on Continuous Integration server as well. So what’s the problem?

Xcode 10 Parallel Testing Issues

Although, parallel testing seems to be working in the above example. It has some issue that it can not be used in the real projects at least for me. The major blockers adopting parallel testing are

• Parallel tests introduced flakiness in the tests. It’s not guaranteed that the test will always pass when run in parallel. It’s true for the example mentioned above in the demo app.
• When one of the tests fails, parallel test waits for a long time, resulted in the long build time. I made a test to fail intentionally and it resulted in 7 min build. The build log for the failed test can be found here
• Debugging the test become harder as we can’t reproduce the failures on the local machine.
• The test reporting tools like xcpretty don’t work well with the parallel test so we lose the nice output in the build logs. Well, this is a limitation of xcpretty
• Fastlane scan doesn’t have inbuilt options to run the test in parallel so we have to add additional parameters it as xcargs and disable xcpretty formatter to get this working on CI.
• While running the test with xcodebuild  we need to add -quiet  option otherwise we get massive build logs. On Travis CI, they got a limit of 4 MB logs . Also when the test fails build logs waits for more than 10 minutes causing Travis to build to terminate.

There are some of the common issues observed while trying parallel test feature but the main blocker is flakiness. Until flakiness is there, it doesn’t make sense to adopt the parallel testing feature of Xcode 10.

Back to Apple

With this experiment, it’s clear that the parallel test feature is flaky and can not be used until we can resolve the flakiness. The team of Apple engineers working on developer tools might clarify the root cause of flakiness or some of the issues to make the tests stable. Do companies need to but expensive iMac Pros to run UI tests as shown in the WWDC demo?  Or something else to get rid of the flakiness.

There is a demo repo on Github to reproduce the issues mentioned in the tutorial.

Conclusion

The parallel testing feature clearly brings some benefits in terms reducing the build time but it introduced flakiness in the tests especially in the UI tests. Hopefully, Apple will fix or prothe vide workaround to get the stable results from the tests. What are your experiences of Xcode 10 parallel tests? Share in the comments below.