•
Drizz raises $2.7M in seed funding •
•
Featured on Forbes
•
Drizz raises $2.7M in seed funding •
•
Featured on Forbes
Your mobile application could pass every local test and still have a broken login screen or crash on specific versions of Android. Automated tests can help, but their placement in your release pipeline is equally important as the test coverage.
Match each test to the right stage of the software development lifecycle. Run unit tests as developers build and commit individual functions. After a build, use smoke tests on an emulator to check app launch and login. Run regression tests on physical devices to confirm real-world behavior, then monitor production for crashes and failures after release.
Testing out of order wastes time. Unit tests cannot show how a real device handles camera permissions, while running a full device suite on every commit slows development. Finding a login failure after release creates avoidable rework.
So the right combination of speed, accuracy, and risk with quick feedback and coverage is crucial. Run smaller tests early in the build process. Test the app and make sure major user flows work prior to release. Define a point at which a failure should halt the process and communicate to the development team when a build is ready for release.
In this article, we are going to discuss where automated tests belong in the release pipeline for a mobile app.
5 things make mobile pipelines harder to build and maintain.
A challenge when using CI for mobile apps is building iOS and Android apps. The build environments for the two mobile operating systems are significantly different. For iOS, the build environment is specific to macOS and Xcode.
For Android, the build environment is specific to the Android SDK and Gradle. You can't build an iOS app on a Linux CI runner. If you want to use GitHub Actions for building iOS apps, you need macos-latest runners. These runners are more expensive and slower than Linux runners. Bitrise and Appcircle are two CI/CD tools for mobile apps that exist because generic CI tools handle mobile builds poorly, and they provide out of the box solutions for building and integrating mobile apps.
With iOS development, every build is signed with a certificate and a provisioning profile. The same concept is applied to Android releases, which are signed with a keystore. Managing these certificates and keystores across a team and the CI environment is where most first-time mobile CI setups fail.
Fastlane is a mobile automation tool which provides many solutions to challenges faced in mobile development, and match is one of them. It provides a solution for code signing in iOS. It is useful for storing codesigning certificates in a private Git repository. However, it requires initial setup and certificate revocation.
Testing on the web covers 3-4 browsers. Testing on mobile covers Samsung (One UI), Pixel (stock Android), budget devices (3-4 GB RAM) and iPhones. Emulators cover some bugs. OEM specific bugs show only on real devices.
Web deploy goes live in seconds. Mobile deploy goes through App Store Connect/Google Play Console with review that takes hours to days, and gradual rollout. There's no way to quickly fix a production bug by pushing a commit. You submit a new build and wait.
Web apps are served via HTML and JS from a CDN. Mobile apps are binary (.ipa, .apk, .aab) and need to be installed on the device. Testing a new build of a mobile app requires distribution via TestFlight, Firebase App Distribution, Google Play internal testing, or sideloading.
The mobile CI/CD system automates code integration, build, test, and deployment. With continuous delivery, a release is made ready for approval. With continuous deployment, enabled by platform rules, a release is automatically performed.
The mobile CI/CD process integrates platform CI/CD tools, signing keys, differences in devices, and review by the app stores. Building iOS software requires macOS and Xcode, and building Android applications requires the Android SDK and Gradle. Physical devices introduce differences due to memory, graphics, permissions, and software by the manufacturer.
As users install app updates at different times, the backend systems are required to support older versions of the app. Halting a rollout means the following copies cannot be made available for distribution. These considerations necessitate that pre-release checks are more thorough.
Here are the different stages where automated tests belong:
Each pull request should contain lint, type check, and unit tests. These fast checks can be performed even before a signed release. Some of the tests might even require compilation or use of a simulator.
Unit tests can evaluate business rules such as the calculations of discounts, validate inputs, check the expiration of tokens, and so on. Contract checks of the API should be put in place for shared requests and responses, especially if changes behind the mobile client impact older mobile clients.
These checks must be performed prior to merging to confirm that the checks succeed. The output of the checks which have failed must include information about the broken rule, its associated module and the commit which has made the specific change.
After all checks have passed, build your app with the required release configuration. Fix your dependencies. Put your signing keys in your CI secret manager. Fix your toolchain versions.
Verify certificate signing, app build identifiers, entitlements, version number, and environment settings. Distribution should be disabled by an expired certificate and a build configuration should fail for a production build that builds against the staging API.
Keep the intermediate artifact and its build ID that was created in this stage. This helps you provide support for the following stages. Building from the same commit but with different configurations or dependencies generates a different artifact and invalidates your earlier testing.
A debug build must not be accepted as a release build. For instance, one could shrink Android code which could potentially remove a class that is accessed through reflection. The configuration and the release must be tested by installing the app from the store using an internal or beta release channel.
Conduct a smoke test on the Android emulator or iOS Simulator. Ensure you launch the application and log in (where applicable). Post that, browse the application and place some order in the sandbox.
Assert all outcomes of the flow. Until you validate the order, amount and confirmation, a tap on the “Pay” button does not accomplish anything.
The smoke test should be followed by regression tests for the modified feature and related features. A changed shared authentication will require more extensive regression testing compared to a change of text on a single screen.
Maintain a stable smoke suite on every pull request, even though you choose to execute other tests based on changed files. File-based selection does not capture side effects.
Block progression when an essential flow fails. While emulators and simulators create a predictable environment, passing this stage does not confirm whether the behavior of the actual device is as expected.
Test important flows of the product on real devices of iOS and Android before performing a beta release or publishing the application to the app store. Test the flows that contain permit requests that are rejected, intentional disruption of the network, allow the application to go to the background, and then relaunch the application.
Build a device matrix leveraging data from product usage, OS support data, and old bug reports. These configurations may be required for users of Samsung Android devices with lower memory. For the iOS side of things, test with the oldest OS supported, as well as different sizes of OS displays.
Test upgrades and new installs concurrently. Install the newest version that has been released to the public, create user data, and then test the upgrade. Make sure that the database migrations, user's saved preferences and the session state is maintained. There is not much value in testing whether a new install retains old user data.
Measure the time it takes to start the app and consume system resources. Benchmark the data with your baseline and set limits within a range that is acceptable to test.
These testing tools have an adaptive nature and can help reduce the burden of maintenance for testing frameworks. One example of such a tool is Drizz.
Drizz allows you to write test scripts in plain English. It uses Vision AI to control real devices. Its features also allow you to maintain regression coverage and enable UI self-healing. CI integration allows you to run test scripts and maintain control over the testing process.
Adaptive testing tools must also articulate the rationale of when to end a test. For example, if the checkout button is moved on the page, then the tool must verify the order total and the payment result.
When using a service for testing devices, you need to check the status of each run. Uploading a test or a request does not mean the test was successful. Timeouts, missing results, and especially important tests that are skipped should be considered as incomplete test evidence. Builds should never be promoted until these issues are resolved.
Passing candidates should be distributed through TestFlight, Google Play testing tracks, or Firebase App Distribution, depending on your workflow. Test the installation and upgrades, and also test deep links and push notifications through your intended distribution channel.
Beta testing involves human usability testing and helps with identifying various unexpected issues. It complements automated testing and can never replace it.
If possible, do phased or staged releases. Analyze Android app not-responding and app crashing incidents, log ins and checkout completions by application version. A session where a user exited the app before completing payment should be considered as a failed session even if there were no app crashes during that session.
Set rollout limits based on your existing baseline and specific business goals. Analyze the failure distribution across different device models and the sample size before extending your distribution. Hidden failure rates can exist for specific device models for a particular distribution.
If rollout limits have been exceeded, roll out should be paused and the reason should be investigated. Feature flags can help mask the affected functionality, but only if the appropriate controls have been put in place. Binary defects need an updated build and a new release.
Code commit
Define different test scopes depending on the pipeline trigger. Pull Requests are checked against speed, smoke, and regression tests on the code change. More risk is associated with changes to the Camera or Payment systems and would require a physical device test before merging.
For release candidates, increase test coverage to devices on a regular basis. Overnight results of older builds should not approve a new candidate without adequate testing.
Evaluate the time from commit to test result and factor the time in device queues and test set up. As long as data, accounts, and devices are in isolated states, run tests in parallel on devices.
Maintain a system of flaky tests. Take the first failure of a test and determine whether the test failed or the system crashed. Set time bounds on quarantined tests. If a quarantined test is to log in or make a payment, provide an alternate check to be performed before a release.
If your code is already hosted on GitHub, GitHub Actions is obviously an easy choice since you don't need to take the additional effort of incorporating a different CI tool. It builds for both iOS and Android, but iOS builds require macOS build runners, which are about two to three times slower than Linux build runners. The free tier allows for 2000 build minutes a month, and on macOS each build minute counts as ten.
Jenkins is free to use and allows a lot of flexibility. However, you will have to set it up, manage and maintain it yourself. This means you'll also need to get and manage your own Mac build machines for iOS. This becomes feasible when a team is already using Jenkins and it is owned and managed by a DevOps engineer.
Bitrise is entirely focused on mobile, with native iOS and Android builds and integrations for code signing and device testing. The free tier allows one concurrent build and 30 minutes per build, and paid plans start at $89 a month.
Fastlane is different from the other CI tools. It does not serve as a full CI server. Rather, it helps automate various tasks such as code signing (match), building (gym/gradle), testing (scan) and shipping (pilot/supply). It integrates with various CI servers such as GitHub Actions, Bitrise and Jenkins.
Appcircle is a mobile CI/CD which allows its users to set up the pipeline by dragging and dropping, thus, eliminating the need to write YAML. This is starting to gain traction with enterprise mobile teams. It also manages Apple certificates and provisioning profiles for you.
The CI you choose covers stages 1 to 3. These tools help move a build through the development pipeline, but the workflow still needs reliable validation on real devices. Stage 4 is testing on actual devices, and this is where Drizz’s API comes in. You plug it into the CI of your choice, upload an APK, start a test plan, and get structured, pass/fail results your pipeline can gate on. Casey Nguyen, an Engineering Manager, said “Drizz felt genuinely plug-and-play. We didn’t have to fight setup or spend days wiring things together.”
Start by identifying the failure that creates the biggest release risk. Then look at how your app is actually used, where users are most likely to encounter problems, and whether your testing process reliably catches those failures before they reach production.
The goal is not simply to automate more steps in the release process. It is to build a testing workflow that gives the team confidence that every release behaves as expected on real devices. This is where Drizz fits into the workflow by helping teams continuously validate mobile applications with less test maintenance, so more of the team's effort can go toward expanding coverage and shipping with confidence.
Use fast checks early, validate essential flows before release, and test important scenarios on real iOS and Android devices before shipping. The goal is to balance fast feedback with enough coverage to catch failures that emulators and unit tests cannot.
Mobile CI/CD automates the mobile app development build, test, and deploy process. For mobile CI/CD, the pipeline automatically builds, tests and deploy the mobile apps, if the code changes, builds are triggered automatically, and tests are run across the devices. Passing builds are deployed to testers or app stores without manual intervention.
There are five reasons. Building iOS apps require macOS, code signing involves certificate management, device fragmentation requires testing on multiple devices, app stores have review gates, and mobile apps are compiled code and need to be installed on each device.
Bitrise and Appcircle are specific to mobile. The rest of the tools are general-purpose tools. If your team is using GitHub, then you can use GitHub Actions. Others can use Jenkins, but consider the DevOps costs to maintain Jenkins. Most teams use Fastlane for build/deploy tasks with other CI tools.
Add a stage after your emulator-based tests to upload your build to a real-device testing platform (Drizz or BrowserStack, for example) and test via API. Make this the gating step for your pipeline. This means that if any of the real-device tests fail, the build does not get distributed.
Build: 3 to 10 minutes. Unit Tests: 1 to 3 minutes. Smoke tests on emulators: 2 to 3 minutes. Real-device regression: 15 to 30 minutes. Total: 20 to 45 minutes for a complete pipeline. Smoke-only pipelines (no full regression) can run in under 10 minutes.
It is possible to share tests and distribution logic if your tests are cross platform (like Drizz tests that are written in plain English and are the same for both iOS and Android). Building and signing logic is different (Xcode vs. Gradle and using certificates vs. keystores).
Related Content:
Build triggers in Jenkins | Smoke testing in CI/CD | CI/CD tools compared | Book a demo
Read More:
©Drizz 2026 • All Rights Reserved
