When a platform holding your secrets is breached: secrets rotation runbook

Why this needs a runbook. Modern apps rarely keep their own secrets. Deployment platforms store them as environment variables, CI/CD systems hold publishing and cloud tokens, and AI gateways and developer tools log requests that can contain tokens. When that provider is breached, its notice usually says secrets may have been accessed, not which ones. You cannot wait for certainty: the safe working assumption is that everything the provider held is in the attacker's hands.

Recent examples

On 21 October 2026 Railway said an unauthorised third party may have accessed customers' encrypted environment variables and, for some, obtained the decrypted contents, and told all users to roll every environment variable and private registry credential stored with it. The same day LiteLLM said attackers had published poisoned releases using developers' GitHub CLI OAuth tokens, which it believes were taken from the logs of an internal LiteLLM deployment exposed to the internet. Earlier in October, developer host Zeabur reported a breach in which stored environment variables were read. The details and current status are on this desk's Railway, LiteLLM and Zeabur cards.

Rotate, revoke, expire: what each one does

Rotation replaces a secret with a new one. OWASP's Secrets Management Cheat Sheet recommends rotating regularly so that stolen credentials only work for a short time, and automating it, because manual rotation invites mistakes. Revocation invalidates a secret at the system that issued it; OWASP says secrets that are no longer needed or potentially compromised must be revoked. Expiry builds the end date in at creation.

The trap after a provider breach: editing the value stored at the provider does nothing to the old secret. The leaked database password, cloud key or API token keeps working until you change it in the database, cloud account or API service that issued it. Rotate at the issuer first, then update the copy at the provider.

Issuer specifics worth knowing

GitHub. You can revoke a single token, revoke an OAuth or GitHub App authorisation (which revokes every token tied to it), or revoke all your credentials at once from account settings, which GitHub suggests when you believe your account may be compromised. GitHub automatically revokes OAuth tokens and personal access tokens that have not been used for a year, and valid tokens pushed to a public repository or gist. Its credential revocation API lets anyone submit a leaked personal access token (ghp_ or github_pat_) or OAuth app token (gho_) for revocation without authenticating. Your account security log covers the last 90 days and can be exported as JSON or CSV; expired or revoked tokens can show up there as oauth_authorization.destroy.

AWS access keys. AWS's IAM guide describes routine rotation as: create a second access key while the first is still active, move applications to it, set the old key to Inactive rather than deleting it straight away, even if its last-used information suggests it was never used, then delete it once nothing breaks. After a breach, treat the overlap period as risk: make the old key inactive as soon as its replacement is deployed, and check its last-used details and your CloudTrail logs for use you do not recognise.

Secret scanning alerts. GitHub's secret scanning guidance is to rotate an exposed credential immediately; it notes that scrubbing the secret from Git history takes time and is often unnecessary once the credential is revoked. Validity checks can tell you which detected secrets are still live, so you fix those first.

Runbook

1. Scope it. List every secret the breached provider held or could see: environment variables, build and deploy secrets, registry and package-publishing tokens, webhook signing keys, OAuth app credentials, and tokens that may have reached its logs. Do not limit the list to what the provider says was affected; their own scoping is usually still under way.

2. Rank by blast radius. Rotate first what an attacker can use from the internet without anything else: cloud provider keys, package-publishing and source-control tokens, payment and messaging API keys, and AI inference API keys (Railway says the series of attacks it believes it was hit by aims to compromise AI inference providers). Then database and internal service credentials.

3. Rotate at the issuer, then redeploy. Create the new secret at the issuing system, update it at the provider, redeploy, and only then disable the old one. If the provider has paused deployments (Railway did on 21 October), decide case by case whether to revoke immediately and accept an outage, or wait for redeploys.

4. Hunt for use. Review the issuer's logs for each old secret from the earliest possible exposure time: GitHub security and audit logs, cloud audit logs, package registry release history, and API usage dashboards. OWASP recommends keeping CI/CD logs queryable for at least 90 days because working out how a secret was taken can take time.

5. Remove the reasons it could leak. Shorten lifetimes and prefer credentials that are issued per job. PyPI's Trusted Publishers, for example, let a CI service exchange a short-lived OIDC identity token for a short-lived, project-scoped upload token, so the CI job never needs a long-lived PyPI token. Do not log request or response bodies from AI gateways or developer tools unless you must, and keep those systems off the public internet.

6. Record and report. Keep a list of what was rotated, when and by whom. If the hunt shows personal information was accessed, start the NDB assessment clock (see the NDB clock page); critical infrastructure entities should also check their SOCI reporting duties.

Practitioner checklist

Could you, today, produce a list of every secret stored with each of your platforms and rotate all of them within a working day? If not, fix that now: keep an owner and issuer for each secret, move as many as you can to short-lived or per-job credentials, and practise a rotation before you have to do one under pressure.