Quality coverage for a mobile SDK at 200M+ user scale

An SDK embedded in other companies' apps has no safe place to fail, and integration risk, device behavior, and API reliability all meet in it.

This is the shape of a case study, with the specifics kept at a level I can share. Replace the placeholders with real numbers you are comfortable publishing.

The constraint

A mobile SDK ships inside someone else’s app. When it breaks, the host app’s users blame the host app, and the host app’s team blames the SDK. The products it served added up to more than 200 million monthly active users, so a defect that looked small in isolation was never small in effect.

What I tried first

The first instinct was to cover the SDK’s public API surface with unit and contract checks and call that the safety net. It caught real bugs, but the failures that hurt were never in the SDK alone. They lived in the seams: a host app on an old Android version, a permission dialog appearing at the wrong moment, an API response the SDK had never seen in a lab.

What it became

Coverage moved to where the risk actually lived. Appium suites exercised the SDK inside realistic host apps on real iOS and Android devices, not just simulators. API checks validated the contracts the SDK depended on, so a backend change that would break integrators surfaced before release. Device, OS version, and network condition became explicit dimensions of the test plan instead of luck. Along the way I mentored the QA engineers who kept the suites alive after me.

Where it landed

• Integration defects surfaced in the pipeline instead of in partner apps, which is the only place an SDK team can afford to find them.
• Release conversations moved from “did we test it” to “which risks did we cover and which did we accept”, which is a different job.
• The suites outlived my involvement because maintaining them was part of the design, not an afterthought.

The honest tradeoff: real devices and realistic host apps are slower and more expensive than simulators. At SDK scale the cost of being wrong is someone else’s outage, so the trade was easy to defend.