WWDC18: What’s New in Code Coverage, XCTest and XCUITest

Published by Shashikant Jagtap on

At WWDC18, there was a session on What’s New in Testing describing new features in Code Coverage, XCTest, and XCUITests. It was presented by the Honza and Ethan. This session consists of following

  • xccov  for Code Coverage
  • Test Selection from Scheme
  • Randomising Tests
  • Parallelising Tests
  • Tips for Parallelising Tests

Fortunately, I have already covered most of the things presented at this talk in my previous blog posts.

  • As xccov has been launched with Xcode 9.3, I have written a detailed blog post covering all the features of xccov on my personal blog as well on Medium.
  • Parallel Testing features of Xcode 10 has been announced in the Xcode 10, and I already covered most of the parallel testing in Xcode 10 in action blog post which is also published on Medium

You can always watch the live demo in the session anytime. However, we will cover the features mentioned at the session briefly.

Code Coverage

Apple has released new command line tool xccov with Xcode 9.3 for inspecting the contents of Xcode code coverage reports. We can explicitly enable the code coverage for the scheme by editing the scheme and tick the box ‘Code Coverage’ from ‘Test’ action.

 

Once the tests run with code coverage data, Xcode will generate Code coverage reports into the default derived data directory located at ~/Library/Developer/Xcode/DerivedData and you will see the code coverage reports generated at Logs/Test directory. We will see code coverage with .xccovreport  and .xccovarchive  extension. Inside the Logs/Test directory, there is coverage report, with extension .xccovreport, and the coverage archive, with extension .xccovarchive respectively. As per man page of this utility “The coverage report contains line coverage percentages for each target, source file, and function/method that has coverage information. The coverage archive contains the raw execution counts for each file in the report”. We can achieve following things with xccov at the moment

  • View the Code Coverage Reports from Terminal
  • Spit out the JSON from code coverage reports.
  • List all the files for which code coverage has been generated
  • Look at the code coverage report for one specific file.

Watch how to view code coverage in action.

 

This reduces the pain of using third-party tools to display the Xcode code coverage reports in the nice format. We can also skip the code coverage for the targets which we won’t care.

XCTest Scheme Options

With Xcode 10, We can select the specific tests from schemes, randomize the tests and execute the tests in parallel. We can enable the parallel testing by updating the scheme and in the Test action, we can select “Options” against test bundle to choose parallelization option. We can select location as well.

 

The tests get extended to parallelizing test suites within single simulator by creating clones of the simulators. Xcode creates different runner process under the hood and each process gets specific tests assigned.

It’s possible to run the XCTest from the command line using the xcodebuild tool. With Xcode 10, we have few more options to enable parallel testing those are the

  • -maximum-concurrent-test-device-destinations NUMBER : the maximum number of device destinations to test on concurrently
  • -maximum-concurrent-test-simulator-destinations NUMBER : the maximum number of simulator destinations to test on concurrently
  • -parallel-testing-enabled YES|NO : overrides the per-target setting in the scheme
  • -parallel-testing-worker-count NUMBER : the exact number of test runners that will be spawned during parallel testing
  • -maximum-parallel-testing-workers NUMBER : the maximum number of test runners that will be spawned during parallel testing

These options allow us to execute the XCTests in the parallel as we have done it from Xcode 10.

Tips for Parallelising Tests

At the end of the session, there are three awesome tips shared which are

  • Keep the test class small, if the tests class getting big with too many tests then split it into 2 or more classes. This will avoid the situation of one class getting dominated on the parallel testing
  • Keep the performance tests in the separate bundle and never run them in parallel
  • Find the right tests for the parallelization

Conclusion

There are some awesome improvements happened this year in the parallel testing of both XCTest and XCUITests. There is one more session is going to happen on Testing Tips and Tricks where we will get more insight on how to perform testing better. I will publish another post about that. Stay tuned.