CIO Brief: When a Dev Tool Breach Forces a Company-Wide Credential Reset
Retrospective: this article looks back at events from January 2023, written in 2026 with the benefit of hindsight.
The short version: In January 2023, CircleCI — a service many companies use to build and deploy software — told every customer to change every password and key stored in it. For many companies, that meant days of emergency work.
Why this was so disruptive
Build and deployment tools hold the keys to production: cloud access, databases, payment APIs, code signing. Changing all of them at once, without breaking anything, requires knowing where every key is used. Many companies didn't.
The business impact
- Emergency engineering work across teams.
- Risk of outages from rushed changes.
- Uncertainty about whether stolen keys were used.
Questions to ask your team
- If our build platform told us to rotate all secrets tomorrow, how long would it take?
- Do we have an inventory of where our secrets are stored and used?
- Have we replaced stored cloud keys with short-lived access where possible?
- Have we ever practiced a full rotation?
What good looks like
An inventory of secrets with owners, documented rotation procedures, a practiced drill, and fewer secrets overall thanks to identity-based access.
The decision
Fund a secrets rotation drill this year. It turns a future emergency into a routine task — and usually reveals forgotten keys worth removing.
- CircleCI Secrets Breach (Jan 2023): Rotate Everything Incident Teardowns
- How to Run a Secrets Rotation Fire Drill Across AWS and Azure How-To & Hardening
- Detecting CI/CD Secrets Theft: Sentinel and GuardDuty Detections Detection & Response