Background
CircleCI is a popular continuous integration and delivery (CI/CD) platform that helps developers automate building, testing and deploying software. With customers in the tens of thousands, it is a widely-used DevOps tool that helps companies such as Google, Peloton and Asana quickly deploy new versions of their products.
On January 4th, 2023, CircleCI informed all of their customers of a security incident and asked them to rotate all secrets stored on CircleCI’s systems, such as cloud provider credentials and repository SSH keys. From the perspective of CircleCI customers, such a security incident is equivalent to the theft of API keys in different production environments. For example, some customers reported misuse of their AWS accounts using AWS credentials stolen from CircleCI.
On January 13th, 2023, CircleCI published that the root cause for the breach was an employee’s Mac laptop that was infected with custom-made malware. CircleCI’s EPP/EDR/AV software did not detect or stop the malware. The malware was able to steal a cookie containing a post-2FA token that was used to access the CircleCI production systems. By using the token, the unauthorized 3rd-party managed to exfiltrate encrypted customer secrets, including AWS keys and GitHub OAuth2 tokens, and had the key to decrypt them.
The Human Element
CircleCI did not disclose how the malware appeared on the laptop, but according to the disclosed information, the malware probably posed itself as a “PTX Player” for Mac. PTX files are a type of transcript file format that is commonly used by educational institutions and other organizations to store and share transcripts electronically. If we had to guess, a CircleCI employee was targeted via a sophisticated spear-phishing/social engineering attack convincing the employee to download this app in order to view/sign some “important” PTX document.
This demonstrates (yet again) how email is still the easiest way into an organization, even into tech companies that are security-aware and invest heavily in cybersecurity. Recently we have seen a rise in similar incidents, resulting in customer data exposure or theft. A few recent examples:
- A social engineering campaign against MailChimp employees and contractors led to password theft and customer data breach
- A LastPass developer’s endpoint was compromised with malware, as part of the latest data leak and customer data was stolen
- A phishing email campaign targeted an employee at The Guardian and the personal data of employees was stolen
These incidents all have one thing in common: an employee or contractor was tricked into either providing credentials or running malware on their devices – and the results can be dramatic. This was achieved by either a phishing e-mail or other means of tricking the target, or victim, to visit a malicious website. This should be a wake up call to security teams that email protection is a key cybersecurity layer that must be a high priority – CISOs should not settle for standard/basic email protection and instead seek out advanced email security solutions.
DevOps Platform Security Tips
Back to the CircleCI incident, if your organization was also impacted, make sure to follow CircleCI’s official recommendations.
Even if you’re not a CircleCI customer, we highly recommend considering the following security measures related to secrets stored on your CI/CD and DevOps platforms:
- Use short-lived temporary credentials. For example, OpenID Connect-based authentication supported by multiple DevOps platforms.
- If temporary credentials cannot be used, automatically rotate non-temporary credentials periodically and on demand.
- Take advantage of IP ranges provided by your DevOps platform to limit inbound connections to their systems. If a token is misused outside the platform, it would be easy to track.
- Run the DevOps execution environment in your own servers directly to inject secrets without storing them in the DevOps platform. This decreases your exposure.
- An internal audit of all stored API keys should be done periodically to further decrease the risk.
Regardless of secrets management, it’s critical to invest in the security of endpoints accessing sensitive/privileged infrastructure and production data, as the next breach might involve other elements which are not just DevOps platform secrets but other crucial customer-related data that shouldn’t get into the wrong hands.
