MWS Cloud removed compromised eMudhra roots from CI and cloud infrastructure

MWS Cloud described how, after the private keys of eight eMudhra intermediate certification authorities were published, it removed the associated trusted roots from its cloud infrastructure. The practical lesson for CI teams is simple and not especially pleasant: an updated certificate bundle on the host does not mean that containers, runtimes, and cached images have stopped trusting the compromised authorities.

According to the company, the incident became urgent on the evening of 20 October, when a link to a GitHub repository containing private keys appeared in the team chat. MWS announced a deployment freeze, and later extended the restrictions to CI/CD as well. The work lasted a week: trust had to be checked at different levels, from nodes and base images to JVM stores and separate certificate bundles in libraries.

According to MWS, the root cause was an error in the key-generation script: in April 2025, the private key turned out to be a number from 0 to 255, rather than a random 384-bit value. Researchers were able to recover the keys from public certificates. Eight intermediate CAs under the emSign brand, used for TLS and code signing, were at risk.

Why blocking in browsers doesn’t close the issue

Browsers received blocks through their own mechanisms, including CRLSet and OneCRL; Certificate Transparency also plays an important role for Chrome. But many development tools don’t use the same revocation channels. They check certificates against a local set of trusted roots, which may be in the system, in a container, or embedded in an executable.

So the absence of eMudhra certificates on your own site is not a sufficient check. If a client trusts emSign ECC Root CA — G3 or AAA Certificate Services, it may accept a chain built using a compromised key. MWS lists curl and wget, Docker and containerd, npm, pip, Go binaries, JVM, gRPC, and Terraform providers among the potentially affected components. Depending on where such a client sits in the CI chain, the result could be a dependency being substituted during download.

The scenario the team considered starts with traffic from a runner to a registry or package repository being intercepted. If the TLS client trusts a compromised root, it may accept a forged certificate; the runner then downloads a modified package, and its installation script runs in the build environment. MWS separately notes that an attacker would first need a position on the network path. The team, however, assessed not only the likelihood but also the possible consequences for secrets and subsequent builds.

This is where the convenient model “certificate bundle updated — so everything’s fine” falls apart: one container image may contain five or six independent trust stores. Manual inventory gets you about as far as the first service that reads a different file.

Where they looked for trust and what they changed

The team identified two roots to remove from the relevant bundles: emSign ECC Root CA — G3 and AAA Certificate Services, also known as Comodo AAA Services root. A second trust path went through Sectigo cross-signatures, so checking only the main chain wasn’t enough. MWS reports that it checked 12 test chains after removing the roots; at the same time, it warns that without AAA, some old legitimate Sectigo chains may no longer build.

Updating ca-certificates on the nodes was only the first step. containerd and dockerd keep certificates in memory after startup, so they were restarted after the system store was changed. New nodes needed fixes too: the team updated the golden image and node-preparation procedures, otherwise a host could come up with the old set of roots already in place.

Most of the work was around CI. MWS counted about six hundred runners of six types, some belonging to separate teams. Updating the host did not fix a job container: its TLS requests continued to use the image’s store. As a temporary measure, the runner configuration was updated to pass the host’s system bundle into containers, and variables were configured for individual clients.

This workaround has limits. The mounted certificate directory conflicted with the location of cacerts in Temurin on Debian and Ubuntu: the symlink ended up with no target file, and Java processes failed. Mounting only the bundle file did not break Java in the same way, but it prevented images from updating the store or adding their own corporate CAs. So MWS kept this method temporary and also added a way to build the Java store from the current bundle.

Less obvious copies of the roots turned up too. These included certifi in Python and a vendored copy in pip, botocore’s own CA file and boto2, the gRPC system root file, as well as NSS and Java cacerts. So fixing the system package by itself does not change files shipped with a library or runtime. Some stacks need updated versions; others need to be configured to use the system bundle or need their store updated separately.

The article lists specific fixed versions: Go 1.27.3 and 1.26.10, Node.js 22.23.4, 24.22.1 and 26.12.1, certifi 2026.10.19 and NSS 3.131.1. For JVM, it describes two options: a fixed JDK or blocking certificate fingerprints through blocked.certs. Already-built Go binaries needed to be rebuilt, while, according to the article, fixes were not planned for earlier Go branches.

Images, tags, and verifying the result

Rebuilding images created a separate dependency problem: the fixed build image itself first needed a fixed base. The team released the first fixed images under the previous tags, then under new ones. Because of node caches, a tag did not guarantee that the new content would actually run. MWS declined to enable imagePullPolicy: Always at scale, to avoid creating a simultaneous load on the registry, and instead cleared caches selectively. The check covered more than two thousand hosts; an old image was found on six.

For verification, the team did not rely on a file disappearing from a known directory. A fix was counted when a specific service rejected a test TLS chain. For images, they used a checker that ran the image and looked for unwanted roots in known stores. At the same time, the authors warn that a simplified search can produce false positives, for example on a JVM where a fingerprint list closes the issue, or on the pip package in Debian and Ubuntu, which uses the system bundle.

In conclusion, MWS links the long-term fix to centralized management of trusted roots, image inventory, and having an owner for every runner. The team also describes moving builds to proxy repositories, using an internal registry for clusters, and using new tags for fixed images. For a quick initial check, the authors recommend running a TLS test from the CI job itself: the state of a laptop doesn’t show which store the build container trusts.