A mobile app is tested on phones. That does not work for a product running on someone else’s hardware: every manufacturer has its own platform, firmware, and system version. A build that reliably brings up a session on one configuration may not start at all on another. Checking every release by hand on every platform is impossible: there are too many combinations. The sequence worked out in automotive software testing carries over to IoT, embedded, and mobile development.
It comes from software engineer Maksim Lykov, a speaker at Conf42 IoT and the Worldwide Software Architecture Summit, who worked on launch reliability for Android Auto.
Benches Are Assembled from Standard Hardware
Most often that hardware is close to what the manufacturers themselves use. With Android Auto, that means head units to which a phone connects over a wireless channel.
Covering the full range of platforms at once is expensive and scales poorly. Teams start with a few standard configurations and run them constantly.
Standard hardware also sets the boundary of the task. What gets tested is your own software, and the question is: can the launch be made more stable on your side, with no changes at the manufacturer?
The answer defines what counts as a finding at all. A defect the team can close on its own is worth more than one that will take months to negotiate.
The automation is written in a language convenient for driving devices. In the case described, that means Java and ADB commands, the standard tool for controlling an Android device from a computer.
Why the Runs Do Not Fit into a Normal CI
A bench with real hardware does not fit into the ordinary build pipeline. Benches were updated regularly with recent builds, and continuous runs followed. Several benches work simultaneously.
The arrangement has a cost. Rare failures made validation extremely slow: when a problem appeared only occasionally, several passing runs were not enough to show that a fix worked. The bench sometimes had to run for many hours before the statistics became meaningful.
In exchange, the team gets an uninterrupted series of identical launches on the same hardware. A short CI run gives nothing of the kind.
Upon failure, reports from the device and the bench are saved, along with the automation logs. A separate dashboard shows statistics for each scenario type, and without it, rare defects remain invisible.
Sleep, Reboot, a Second Phone
Many of the harder-to-reproduce defects appeared around state transitions:
- repeat connections;
- launch after a reboot;
- launch after suspend-to-RAM, the sleep mode that keeps the system state in RAM;
- switching between several devices.
Each scenario runs separately, and statistics are kept separately for each. A combined counter across all scenarios will not show the rare failure, and the bench was assembled for exactly that failure.
One Failure in Ten
Some errors occur rarely and depend on the hardware state after previous runs. In wake-from-sleep scenarios, the defect shows up only sporadically.
After a fix, you cannot make a few runs and declare the defect closed: at a rate of one in ten, a series of passing runs is quite likely with no fix at all.
So the stress test is left running for a long time, and the statistics are read afterward. The result is then checked on manufacturer benches when access is available. Logs rarely cause trouble: extra diagnostics are quickly added to a test build.
The metric is counted from bench runs. User-side telemetry helped show where in the connection sequence failures happened, while the bench made it possible to repeatedly reproduce and measure a specific scenario under controlled conditions.
The bench answers a narrower question: whether things improved in the scenarios that run on a loop. A decision about a fix needs that narrow answer.
In Lykov’s experience, over the first month of a bench’s operation, the number of launch errors in repeat tests dropped noticeably.
A Stable Bench and Fallback Paths
A separate requirement is predictability of the test environment itself. If the bench behaves unstably, telling a defect in the product from a defect in the test becomes impossible, and the whole body of statistics loses its meaning.
What to do about defects on the other side is decided separately. The team cannot change the manufacturer’s firmware.
A typical case is when a component, such as the Bluetooth module, appears healthy after waking from sleep but no longer behaves as expected. In such situations, recovery logic on your side can serve as a fallback when the underlying platform cannot be changed immediately.
The way out is recovery logic for the module on your own side. It works regardless of how sleep mode is built on a particular platform. Even if the underlying platform behavior could not be changed immediately, recovery logic on the Android Auto side could provide a fallback, making the connection more resilient.
The product checks the state of an external component before every use, even when it should have stayed the same after sleep.
What is worth building into the bench from the start: more hardware variants, a degraded wireless signal, increased distance between devices, interference from nearby equipment.
The Same Rules in AI Model Evaluation
The approach carries into fields where nothing like an automotive bench exists.
One successful run is too little to call a system working. Testing software on someone else’s hardware and evaluating AI models both require varied scenarios, repeated runs, and checks of behavior under unusual conditions.
The difference lies in the criterion. A session either came up or it did not; the result is unambiguous. With a model, the line between a right answer and a wrong one is blurred, and the main difficulty moves into defining quality itself.
On the bench, the system sets the criterion; in model evaluation, it must be designed separately before the first runs.
Lykov now works on tasks related to evaluating AI models. By his observation, automation in evaluation and training will grow, partly through synthetic data. Companies are actively exploring how synthetic data can be used to scale training while maintaining acceptable quality.
More specialized tools will appear for particular fields, where checking that an answer is correct falls short and the result is judged in the context of real work.
The logic is the same as on the benches. Look at how the system behaves across a series of runs.
Photo By: Фотограф: Nothing Ahead: Pexels
Priya Nandakumar covers enterprise technology and AI infrastructure for DevX, with a focus on the systems decisions that look fine until they don't. Caching layers, message queues, fault tolerance. She spent seven years as a backend engineer at two Series C startups before moving into technical journalism, and she still reads changelogs for fun.
