An IT market has been shaken by the wave of Artificial Intelligent, Machine Learning (AI/ML) and Continuous Integration, Continuous Delivery (CI/CD) practices. The DevOps and CI/CD movement made the drastic change in the roles of software engineers and it highly impacted QA engineers. The testing at the pace of DevOps is impossible using the traditional QA methodologies and QA engineers are struggling to find their role within DevOps workflow. As a result of this, confused QA engineers ended up buying the fully theoretical book on “Practical Testing in DevOps” to get some idea to survive. However, reading the book won’t make them survive in this fast-paced DevOps world. They need to acquire hands-on skills and learn modern technologies to survive in the market. This gives rise AI/ML based testing tools like Mabl hits the market. Mabl currently tests the websites using Machine Learning but it cannot test native mobile applications yet. In this blog post, we will try to find out if any third party AI/ML can test iOS apps in future.
Mabl
Before we dive into testing of iOS apps with machine learning let’s briefly see what Mabl is and how it tests websites using the AI/ML techniques. Basically, a couple of engineers who worked for Google Dan Belcher and Izzy Azeri found the problem that testing in the pace of the DevOps is really hard and traditional approach of testing software aren’t enough in the pace of CI/CD. In order to speed up the testing with DevOps and CI/CD pace, they found the AI/ML driven tool Mabl that will generate the test scripts and run them on Continuous Integration Servers and reports the results. Mabl is script-less, infrastructure-less and virtual QA engineer that tests our web application without the need for managing anything in-house. We can record test scenarios by teaching the bot about your application like you train new QA engineer using the chrome extension. Once we trained the bot then the bot will know more about the application and create and maintains test scripts in the cloud. Mabl can be plugged with many Continuous Integration servers like Jenkins, TravisCI. It also sends test reports to Slack or other services. The Mabl also has insights that come out from the machine learning pipeline. It’s basically machine learning technique that interacts with web element on the page that’s where Mabl differentiates from Selenium WebDriver. The Selenium has issues locating an element on the screen which causes a lot of flakiness in the tests and Mabl will supposedly solve that problem. Mabl is it in the public beta at the moment, you can give it a try. There is a brief article on Mabl here. In a summary, Mabl works like this
You provide URL of your web application to Mabl with login credential if applicable
Mabl analyze your app and crawl for JS errors, broken links etc
You train Mabl using chrome extension and record journeys that will be stored in the cloud. Mabl uses machine learning to analyze your app.
Mabl will execute those journeys on CI server and reports issues
No need to manage infrastructure, code for testing.
Currently, Mabl looks like Record and Playback tool hosted in the cloud but as per Mabl it uses a lot of AI and machine learning under the hood. As of now, Mabl can only test websites and it doesn’t have support for testing mobile apps. As of now, there is no plan to add mobile testing support on the third roadmap as per the answer from Dan in this webinar.
iOS Challenges for third Party AI/ML Tools
Now that we have seen that Mabl can’t test iOS apps at the moment but let’s figure out what are challenges testing iOS apps and will third-party AL/MI can use current Apple technologies to test iOS apps. There might be few challenges dealing with Apple ecosystem. Let’s cover some of them here.
iOS Build and deploy process
In the web world, it’s fairly easy to deploy the app to the specific test server and use URL of that server for testing. However, in the iOS world, we have to consider so many things whilst we prepare test app. In order to prepare debug or ad-hoc version of the app, we need an app to be signed with specific developer certificate and have provisioning profile with a list of devices. it’s questionable that AI/ML tools that will be used have provisioned devices to test app. Another challenger would be prebuilt binary needs to deployed on the devices before actual testing happens. Can AI/ML able to analyse and build an app by themselves we teach them how to do that. There are various moving parts that need to be considered before we get an app in the testable state.
Apple ecosystem
Apple has its own tools and frameworks and they are fairly solid in nature. Apple wants developers to use their tools and technology. It’s possible to reverse engineer Apple frameworks but it’s not worth time and money to do this as apple can break that in the next release. Apple has realized the potential is machine learning and released its own machine learning framework (CoreML) that can be used in the iOS apps. Apple might be also investing the strategies to test the machine learning features effectively. It’s a good idea to leave it to them for now and see what they can offer for us to test machine learning stuff. Efforts made by third-parties to test iOS apps with machine learning won’t be sustainable for a long time.
Training Machine Learning Model
In Mabl, they have built a chrome extension to train a Mabl about your application journeys. It looks like a recorder at the moment. Building recorder for the web apps can be easy but building recorder for iOS apps can be tricky. Apple has its UI testing framework (XCUITest) tightly coupled with Xcode. XCUITest has recording feature that can record the application journeys using the accessibility API of iOS. Can any third party tools enter into the iOS ecosystem and create recorder something like Appium did? How AI/ML will handle the communication between app and trainer without entering into Apple ecosystem? Will that be easy for AI/ML model to record tests by themselves?
There are s many questions about training ML-driven tool to teach all things about your iOS apps.
iPhone Vs iPad support
Mabl just announced the product update on Twitter that it supports cross-browser testing. In the web world once the journies are recorded then it would not be difficult to run them on the different browser using some techniques like user agents or using engines of other browsers. However, iOS has iPhone and iPad with various combination of the screen size with different hardware structures e.g iPhone X, iPhone 7+, iPhone 9, iPad Pro, iPad Mini etc. The question would be once trained AI/ML model for once device can it be applied to all the device varients?. The journey recorded on iPhone will work on iPad or do we need to train the Ml-based model differently for iPhone and iPad. How AI/ML will deal with elements on the screen? Will they use Accessibility APIs or some other techniques to identify elements uniquely without causing flakiness?
Testing iOS App Extensions
Nowadays, its very common to have an extension to iOS apps like Watch Extension, iMessage extension etc. It would be interesting how AI/ML based tool will deal with iOS app extensions. If the AI/Ml based lives in the cloud will it codesign app extensions or pair with iPhone? Will there be real devices for AI/ML tool or it would be just simulators?
It would be very interesting to see how AI/ML will handle this situation.
What’s Conclusion?
Now that, we have seen few challenges in building AI/ML driven testing solution for iOS apps. Considering these challenges, we can say that its possible for third-party to build AI/ML solution like Mabl but personally I think it won’t be sustainable within Apple ecosystem for the long term. Here are things that can be built easily.
Its possible to ask users to upload signed
.ipa file to start testing. Once the binary is uploaded AI/ML based tool will be able to install that in the simulators or devices if they are OK to be provisioned.
AI/Ml driven test will then launch an app and record some journeys and store those journeys in analysis pipeline. It might be challenging how to deal with elements of the app unless they have an access to Accessibility APIs similar to XCUITest.
Integration with Continuous Integration services and third-party services like Slack can be easily achieved AI/ML.
Hosting of the data in the cloud can also be possible.
These things look achievable. However, it would be hard to implement by reverse engineering Apple tools and technologies. So it is not going to happen anytime sooner. The Mabl for iOS still long way to go.
Real Conclusion
Until you are in Apple bubble, there won’t be any Mabl like the third-party tool that comes in and starts testing iOS apps straight away. Apple has released CoreML framework and will definitely release testing strategy for machine learning based iOS apps. Just wait and watch the space what Apple will bring for QA engineers to test iOS apps. Don’t worry about learning Python yet, Apple ecosystem won’t easily allow third-party AI/Ml test tools to enter in their space. But keeps your eyes always open to seeing whats happening around.
Xcode 11 brings some amazing features, the whole list of new features can be found at the release notes of the Xcode 11. Some of the notable ones are SwiftUI framework and integration of the Swift Read more…
Apple’s WWDC conference is bringing some great features for developers. Xcode11 packed up with some amazing features too. Release notes of Xcode have all the latest enhancements. One of the great features announced this year was Read more…
Apple WWDC19 has announced the various things for the users. Some of the software was anticipated but there are some awesome hardware announcements too!. There is a live blog running all over the WWDC19 session Read more…
XCTEQ has stopped business activities until further notice. Our Cookie Policy. AcceptView Policy
