Writing
Anatomy of an Identity Incident

2FA Was On. The Session Walked Out Anyway.

Anatomy of an Identity Incident — CircleCI, January 2023. Malware lifted a 2FA-backed session off one laptop, and a CI platform's vault of customer secrets emptied out.

Michael AbramovichFebruary 20, 20236 min read

Forensic write-ups of public identity breaches, strictly at the identity layer. No vendor is the villain — the point is the structural class of failure, not the logo on it.

The 60-second version

In December 2022, information-stealing malware landed on a CircleCI engineer's laptop. Antivirus didn't flag it. The malware did one thing that mattered: it stole a live single sign-on session — one that had already been authenticated with two-factor authentication — and handed it to the attacker. With that session, the attacker became the engineer, reached a subset of production systems, and from there read what a CI/CD platform exists to hold: its customers' secrets. Environment variables, API tokens, SSH keys, and integration credentials for the systems those customers build and deploy to.

The tell that something was wrong didn't come from inside CircleCI. On December 29 a customer flagged suspicious GitHub OAuth activity, which kicked off the investigation. On January 4 the company told every customer to rotate every secret. The intrusion was one laptop; the blast radius was everyone who trusted the platform with their keys.

The identity timeline

Only identity-relevant events.

  • Dec 16, 2022 — Malware is deployed on an engineer's laptop. It is an information stealer, and it is not caught by the company's antivirus.
  • The theft — The malware performs session-cookie theft, capturing a valid SSO session that had been established with 2FA.
  • The impersonation — Using the stolen session from a remote location, the attacker authenticates as the engineer — no password, no second factor needed, because the session already represents a completed login. They escalate to a subset of production systems.
  • The exfiltration — From production, the attacker reaches stored customer secrets: environment variables, tokens, and keys, including credentials integrating customer accounts on platforms like GitHub and AWS.
  • Dec 29, 2022 — A customer notices suspicious GitHub OAuth activity and alerts CircleCI. This external signal starts the investigation.
  • Jan 4, 2023 — CircleCI discloses, shuts down the compromised engineer's access and broad production access, and urges all customers to rotate their secrets immediately.

The pivot point

The pivot is the moment a 2FA-backed session became a portable object the attacker could carry to another machine.

A session token is the receipt you get after authentication, not the authentication itself. It exists precisely so you don't have to re-prove your identity on every request — which means, by design, it is a bearer credential: whoever holds it is treated as the person who earned it. Two-factor authentication guards the act of getting the token. It does nothing once the token exists, because the token's whole job is to let you skip the login. So when malware copies a live session off a laptop, every factor that preceded it has already been spent. The attacker doesn't bypass 2FA; they arrive after it, holding the thing 2FA was protecting.

That is why "we require 2FA" was a true statement and an irrelevant one here. The second factor was real and it was satisfied. The credential that actually got stolen was the post-authentication session, and nothing about MFA-at-login constrains where that session can subsequently be used.

What would have caught it

Naming controls, not products.

Binding sessions to a device or client posture. A session that only works from the machine and context it was issued to is far less useful when copied elsewhere. This is the control that most directly attacks the pivot: make the stolen token non-portable.

Short session lifetimes and re-auth on sensitive actions. Reaching production secrets should demand a fresh, strong re-authentication, not the coasting authority of an hours-old session.

Behavioral endpoint detection, not just signature antivirus. The malware wasn't caught by AV. Detection that watches for the behavior of session-cookie theft and exfiltration is the layer that sees what signatures miss.

Short-lived, dynamically issued secrets instead of long-lived stored ones. This is the structural lesson. A CI/CD platform that stores customers' long-lived tokens and keys is a vault of bearer credentials; breach it and they're all immediately useful. Moving customers to short-lived, on-demand credentials (for example OIDC-based federation, where the CI job exchanges a short job identity for a momentary cloud credential) means a stolen secret store yields things that are already expiring. The same idea that undid the platform — bearer credentials outliving their context — is the thing to engineer away from your own secrets.

Customer-side rotation that's a drill, not a scramble. The advice "rotate all your secrets now" is only actionable if you can. Knowing where your secrets live and being able to rotate them quickly is a capability you build before you need it.

The last mile

CircleCI built and ran the platform. That part worked as a product. But a CI/CD system is not just a product — it is an identity broker. To do its job it holds, on behalf of every customer, the credentials those customers use to reach their source control, their cloud accounts, their infrastructure. The customer hands those keys over because the platform needs them to build and deploy. Which means the customer's blast radius now includes the security of someone else's laptops.

The last mile here runs in two places. First, between "2FA is enabled" and "a stolen session can't be replayed" — the work of session binding and short lifetimes that the login screen doesn't do for you. Second, and more structurally, between "we store our secrets in the CI platform because that's the easy default" and "the CI platform never holds a long-lived crown-jewel credential of ours." Both spans were ownerless. The platform's responsibility tapered off at the edge of its own systems; the customer had handed over standing secrets and reasonably assumed they were safe; and nobody owned the gap where a bearer token on one engineer's machine could be turned into every customer's keys.

Notice, again, who saw it first. The earliest reliable signal was a customer noticing anomalous OAuth activity in their own GitHub — not the platform detecting its own breach. The organizations that come out of incidents like this in good shape are the ones watching their own identity events closely enough to catch a vendor's compromise from the outside. That capability isn't a feature of the CI platform. It belongs to whoever operates identity for the customer, continuously, across enough surface to know what normal looks like.

For defenders

Five things you can check this week.

  1. Bind sessions to device/posture and shorten their lifetimes, especially for anything that can reach production or secrets.
  2. Replace long-lived stored secrets with short-lived, on-demand credentials (OIDC federation for CI jobs) wherever your platforms support it. This shrinks what a vault breach is even worth.
  3. Run behavioral endpoint detection, and assume signature AV will miss an infostealer.
  4. Build a secret-rotation drill so "rotate everything" is a runbook you've practiced, not a panic.
  5. Watch your own identity events for vendor compromise — anomalous OAuth grants and token use in your accounts are how you find out before the vendor tells you.

Built from public reporting and the parties' own disclosures; sources below.

Sources

#circleci #session-hijacking #ci-cd #secrets #last-mile