Running backend tests as Kubernetes Jobs on EKS

The backend suite needed the same network and identity as the service under test, which the CI runner could not give it.

This is a shape of a case study, with the specifics kept at a level I can share. Replace the placeholders with real numbers you are comfortable publishing.

The constraint

A microservice under test lived inside the cluster and only trusted callers that lived there too: internal DNS, service accounts, network policy. The CI runner sat outside all of that. Running the backend suite from the runner meant faking the environment, and every fake was one more thing that could pass in CI and fail in production.

What I tried first

The obvious move was to open a path from the runner into the cluster. It worked, and it was a mistake. It widened access, it drifted from how the service actually runs, and it turned a test failure into a question of whether the tunnel was the problem.

What it became

Instead of bringing the cluster to the tests, I sent the tests to the cluster. The suite runs as a Kubernetes Job on the same EKS cluster as the service, with the same service account and the same network view a real caller has. GitHub Actions triggers the Job, streams its logs back, and fails the pipeline on a non-zero exit. The test environment stopped being a simulation of production and started being production-shaped.

Where it landed

• The suite now runs against the real network and identity, so a pass means something closer to what a pass should mean.
• Failures point at the code, not at the plumbing between CI and the cluster.
• New backend tests inherit the environment for free, so writing one is a matter of adding a test, not standing up access.

The honest tradeoff: a Job is a heavier unit than a plain CI step, and the first setup took real time. It paid for itself the first time a test caught something a faked environment would have waved through.