Before diving into this article, make sure you take a look at the intro article on iOS CI Olympics game here.

In the introductory post on iOS CI Olympics, we have selected the few player and few games for this competition. We will be evaluating the player one by one on the basis of their performance in the each game.  In this article, the player is BuddyBuild, the very popular mobile continuous integration server.

BuddyBuild

In the iOS Continuous Integration Server Olympics competition,  the first candidate selected is BuddyBuild. The most heard name on Twitter and other tech blogging site, because of its simplicity and ease of usage. BuddyBuild is designed specifically for the mobile continuous integration which supports iOS and Android platforms. The clients of BuddyBuild include some big brands like The Washington Post, FireFox, Slack, Meetup etc. The little bit of the background of BuddyBuild organisation

Company: BuddyBuild
Founded: 2015
Headquarters: Vancouver, British Columbia
Description: A continuous integration, continuous deployment, and user feedback platform for iOS and Android development teams.
Founders: Dennis Pilarinos, Johnny Shi, Christopher Stott
Categories: Developer Tools, Software, Mobile
Website:https://www.buddybuild.com/

In this post, I will share my first impressions of getting started with BuddyBuild for my favourite pet project XCFit using the iOS CI Olympics games.  So Let’s get started with our games

SignUp & Project Setup

The BuddyBuild signup process was very straightforward. I have chosen to use my Github account to sign up which took me straight into the BuddyBuild portal. The process was smooth and easy, the welcome screen wasn’t confusing and explained the next things to do. BuddyBuild live chatters didn’t disturb me as soon as I landed into the portal.

The setting up project requires the access to Github repositories that we want to build on BuddyBuild. We can either give an access to all the repositories or we can pick some of them that we want to build. I have chosen to select 3 of them but our main repo is  XCFit to build on BuddyBuild.

Now that, BuddyBuild has given access to GitHub repositories. The next step is to set up an iOS app on the BuddyBuild platform. We can select one of the repositories to build on BuddyBuild

Once, we select the repository to build then BuddyBuild checkout the source code from the repository. It will then find out the iOS apps, targets and schemes for the selected project.

 

BuddyBuild seems to very smart in searching all the apps and schemes. The XCFit project has two Xcode projects one for the library and another for example app. BuddyBuild did the awesome job in finding out both. We then need to select the scheme or target to build. I selected the XCFit-Example scheme to build on BuddyBuild.

Medal: BudyBuild has earned the Gold medal for the project setup game

 

Default Build Configuration

Now that, we have selected the XCFit-Example scheme to build, then the BuddyBuild started building the scheme and automatically identified the Cartfile  inside the project. BuddyBuild started with running carthage bootstrap  command and then built the dependent project. BudduBuild also ran the tests and archived the test results. The default build execution criteria of the BuddyBuild is very smart and detected all the dependencies frameworks well in advance. There wasn’t any need to script the test execution using xcodebuild  or Fastlane. All these things are automatically handled by BuddyBuild. However, some users may not want the way that BuddyBuild offer at the first time, it would be the waste of the time to run the first build against the default configuration and then customise it. There should be some user interaction to ask users if they want to build with default configuration or not.

Medal: BuddyBuild has earned Gold medal for the default build configuration game.

Build Execution Process

The build execution is very interactive for the BuddyBuild, the logs started appearing as soon as build kicked-off. The build environments like Xcode version and simulator versions, targets, schemes are printed at the start of the build so that user can verify that the build is running on the correct environments.

Also, we can observe that there are separate tabs for Xcode Tests, Code Coverage, and Build Details to get more details of the build. The build execution process for the BuddyBuild is great showing detailed but readable logs.

Medal: BuddyBuild has again earned the Gold medal for the build execution game

Build Artefacts and Test Reporting

Once the build is executed then the build artefacts and log are clearly visible in the associated tabs. The Xcode Tests tab shows the detailed test reports on each device for each target. The further we can also see time to complete the tests and detailed logs if needed.

There is also a tab for the code coverage reports which can be useful for the unit tests. The test reports are also available for different branches so that we can compare the number of new tests added or code coverage percentage from branches. Not sure, how long we can see those artefacts of BuddyBuild logs but they are very informative.

Medal: BuddyBuild has again earned the Gold medal for Build Artefacts and Test reporting game

Real Device Testing

BuddyBuild has an ability to run UI tests on physical devices for Android platforms but I couldn’t find any documentation or setting to configure iOS tests on real devices. As they have already implemented for Android so iOS won’t be very long. BuddyBuild has all the iOS simulator available to use and they have the feature to capture the recording of the UI tests.

Medal: BuddyBuild has earned Silver medal for Real Device Testing game.

Build Customisation

The default build configuration can be easily overridden in the BuddyBuild App settings tab. The build settings have various options to change the environment, scheme, code signing, device and test settings. We can configure our build with the configurations we want to build on. We can also change the Xcode version and test our build with different Xcode versions.

The environmental variables are pretty common in the Continuous Integration system, BuddyBuild has the ability to define the environmental variables with its values. BuddyBuilds provides great options, however,  these options are very limited and probably not enough for your iOS project. You may be looking for the custom scripts but BuddyBuild settings don’t show up the options to write custom scripts or it may be hidden somewhere. BuddyBuild should have more build configuration options or allow users to take control with their own scripts. After spending some time, I found that there is the way to customise the build configuration on this blog post.  This blog describes the workarounds to add custom scripts to build configuration.

Medal: BuddyBuild has earned Silver medal for Build Customisation game.

 

Parallelisation & Concurrent Builds

BuddyBuild has a lot of iOS simulators available for testing so parallelising the tests is just a matter of selecting multiple simulators. However, the concurrent build execution is not cheap. The starter plan of BuddyBuild has only 1 concurrent build as you spend more money,  you will get more concurrent builds but BuddyBuild has an ability to provide you with that concurrency you wanted.

Medal: BuddyBuild has earned the Gold medal for Parallelisation and Concurrent Builds game.

Build Pipelining

Build pipeline allows us to define stages of the build that we can run parallelly or sequentially. BuddyBuild has deep integration to setup build pipelines with BitBucket as mentioned in the documentation, however setting up the pipelines with Github or other source control management are not documented.  Hopefully, this will be possible for the other SCM like GitHub as well but not sure.

Medal: BuddyBuild has earned Silver medal for Build Pipelining game.

 

Infrastructure As a Code

As mentioned earlier, BuddyBuild has a way to create a configuration file with BitBucket but that not sure that can be applied for other SCM. BuddyBuild should have the ability to provide the way to run the same configuration on developers local machine. I think this feature is lacking in the BuddyBuild

Medal: BuddyBuild has earned the Bronze medal for Infrastructure as Code game.

 

 

Directory Caching: CocoaPods & Carthage

CocoaPods and  Carthage are the two most used dependency manager for iOS projects. Both dependency managers check out the dependency source code and build the frameworks in Pods and Carthage directory. This is a very time-consuming process to do for every build.  We need to cache those directories to speed up the build execution. At the time of writing this post, there was no support for caching the CocoaPods or Carthage directories. Someone asked to support roam for caching but BuddyBuild suggested to write the custom script to do this as per this forum.  If users have to write everything with own script then it loses the point of using paid CI services. This is an essential feature and BuddyBuild should make some effort to automate this as they did it for running tests and automating deployments.

Update: After talking to BuddyBuild Team, They have an excellent mechanism to caching both CocoaPods and Carthage. The detailed blog post on this will be coming soon.

Medal: BuddyBuild has earned Gold medal forDirectory Caching game.

 

 

Code Signing & Distribution

The code signing with BuddyBuild seems to matter of giving an access to Apple Developer Portal or upload the certificates to BuddyBuild. I think everything else like downloading provisioning profiles and code signing apps has been taken care by BuddyBuild. However, code signing features haven’t been tested as part of the XCFit project. BuddyBuild seems to have some handy features like auto-provisioning of devices, deploy with iOS certs. BudyBuild can distribute an app to different platforms as well but again you have to make some efforts on writing the custom scripts. The good thing is BuddyBuild has documented deployment process for third-party services here

Medal: The areas that I haven’t fully tested with XCFit projects, I will be giving the Silver medals by default untested but available features.

Third Party Integrations

BuddyBuild doesn’t have too many third party integrations but it covers the mostly used integrations. The App Settings tab has Integrations tab on the left which shows all the integrations e.g Apple Developer Portal, GitHub, JIRA, Trello, Slack, HipChat etc

BuildyBuilds has few third-party integrations but they are probably not enough considering the large range of the open-source projects and services available in the market. The only way to add third part integration is to write your own script as mentioned in this blog post.

Medal: BuddyBuild has earned the Silver medal for the third party integrations game.

 

Speed

BuddyBuid has been super fast in starting the build as well as processing the build. During my trial, the build kicks-off as soon as commit has been made without any delay. The process of setting of the environment, checking out source code, scanning the source for apps, running the build was very quick. BuddyBuild is also transparent in showing the time taken by the build, test and other stuff.

Medal: The BuddyBuild has earned the Gold medal for speed.

Secret Store

BuddyBuild has support for environmental variables, device variables and secure files. This makes the storing of sensitive information more secure. BuddyBuild also has clear documentation and usage information here. You can find the variables in the App Settings in the BuddyBuild dashboard.

 

Medal: BuddyBuild has earned the Gold medal for Secret Store game.

Xcode & Apple Developer Tool Upgrades

BuddyBuild is super quick in the upgrading to Apple developers tools. The great example would be Xcode 9 GM upgrade. As per this tweet BuddyBuild team upgraded to Xcode 9 GM in the just 4 hours after release.

Medal: The BuddyBuild has earned the Gold medal for fast Upgrades of Apple developer tools.

 

 

Open-Source Project Support

BuddyBuild is totally free to use for the open-source projects. This is the reason, I can able to try and use BuddyBuild and it’s awesome features. There is a free plan for starter which allows one concurrent build with 20 minutes of build time. Thanks BuddyBuild for supporting open-source community.

Medal: The BuddyBuild has earned the Gold medal for Open-Source Project Support.

 

 

Learning Curve

BuddyBuild is very straightforward to learn as everything is clear from the user interface. Not only iOS developers but also project managers and non-technical people can use BuddyBuild due to its simple interface. However, there might be some cases where you need to write your own custom script then you have to learn how to integrate third-party tools with BuddyBuild.

Medal: The BuddyBuild has earned the Gold medal for Learning Curve.

Documentation & Support

BuddyBuild has awesome documentation page as well as they have forums to discuss issues.  BuddyBuild also offers support over email and social media. I am really impressed with their prompt response on social media and email. I could offer more than gold medal here

Medal: The BuddyBuild has earned the Gold medal for Documentation and Support.

Clean UI/UX

BuddyBuild has clean and intuitive UI that helps users to set up the projects easily and without any pain. The features are easy to discover with the clean UI.

Medal: The BuddyBuild has earned the Gold medal for Clean UI/UX.

 

Price

The BuddyBuild seems to be free for the open-source projects which s amazing. They got 3 paid plans at the time of writing this post.

So, you have to check your pocket or your bank account before going for the BuddyBuild for sure. There is a starter plan which might be suitable and affordable for some small iOS teams.

Medal: BuddyBuild has allowed me to run builds for free which is great but the pricing plans for enterprise builds are challenging. I am giving Silver medal for the price game.

 

 

BuddyBuild In a GIF

The entire process of getting started with BuddyBuild has been captured in a GIF

BuddyBuild: Pros and Cons

As of now, we have seen the features of the BuddyBuilds in brief, now the time is to decide what’s good and not good with BuddyBuild. There are certainly some great features that BuddyBuild offers but there are some limitations as well.

Pros

The benefits of using BuddyBuild are as follows

• Getting started with BuddyBuild is very easy and painless
• BuddyBuild understand the iOS projects very well from Source Code
• It can automatically build, test and archive and iOS app without any configuration,  provided that we have given an access to Apple Developer Portal
• BuildyBuild has an awesome feature of auto-provisioning devices
• BuddyBuild is super fast in starting and executing the iOS builds
• Test and crash reporting in the Dashboard
• No need to configure or Script the build phases using xcodebuild or Fastlane tools in the source code

Cons

There are some pitfalls using BuddyBuilds which are as follows:

• The team became heavily dependent on BuddyBuild
• Automatic configuration takes control over the tasks performed on CI.
• GUI based manual build configuration which is hard to repeat and recreate for other projects
• Custom scripts feature isn’t easily discoverable.
• Not Many integrations available with third-party tools. It requires additional efforts from engineer to write custom scripts.

XCFit: Build Details

The build logs executed for the XCFit project are available here. We can navigate to details, logs and test tables to get the more information about the build.

BuddyBuild Medal Summary

[table id=3 /]

BuddyBuild medal counts looks impressive. In the rest of the Olympics competition, we will see how other players will perform. We will publish the final medal table at the end of the competition. In the next post, we will see the performance of the Bitrise. Continue Reading