The leaked key was not in the production repository. It was in a forgotten demo branch, copied into a README during testing, pushed to a public fork for five minutes, and then deleted. That was long enough. Automated scanners found it, the cloud bill spiked overnight, and the team learned a hard lesson: deleting a secret from Git is not the same as revoking it.
Secrets management sounds like a large-company problem, but small teams are often more exposed. They move quickly, share access in chat, store keys in local .env files, and let CI/CD systems accumulate powerful tokens. The fix is not bureaucracy. The fix is a simple lifecycle: create secrets safely, store them centrally, grant least privilege, rotate them, monitor usage, and revoke them when they are no longer needed.
Know What Counts As A Secret
A secret is anything that grants access. API keys, database passwords, SSH private keys, OAuth client secrets, signing keys, webhook secrets, cloud access tokens, session secrets, encryption keys, recovery codes, and service account credentials all count. If exposure would let someone read data, spend money, impersonate your service, or deploy code, treat it as a secret.
Common secret types: - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY - Google service account JSON - Azure client secrets - Stripe secret keys - GitHub personal access tokens - Database connection strings - JWT signing secrets - SSH private keys - Webhook signing secrets
Stop Putting Secrets In Code
The first rule is simple: repositories should not contain live secrets. Use environment variables for local development, a secrets manager for shared environments, and CI/CD secret stores for pipelines. For production, prefer cloud-native secret managers or a dedicated platform such as Vault when the operational maturity is there.
# Bad: hardcoded secret STRIPE_SECRET_KEY = "sk_live_..." # Better: load from environment or secret store STRIPE_SECRET_KEY = os.environ["STRIPE_SECRET_KEY"]
Environment variables are not magic. They can still leak through logs, debug pages, crash dumps, shell history, and process inspection. They are a delivery mechanism, not a complete secrets strategy.
Use Different Secrets Per Environment
Development, staging, and production should not share keys. A staging leak should not become a production incident. Give every environment separate credentials, separate permissions, and separate rotation schedules.
Environment boundary example: dev app -> dev database, dev API keys, low quota staging app -> staging database, test payment keys prod app -> prod database, live payment keys, restricted access
This also makes incident response easier. If a development token leaks, you revoke the development token and investigate. You do not have to rotate the entire company.
Add Secret Scanning Before The Incident
Secret scanning should run in three places: developer machines before commit, pull requests before merge, and the repository platform after push. GitHub secret scanning, pre-commit hooks, and tools such as Gitleaks or TruffleHog can catch many accidental exposures.
# Example: run gitleaks locally before pushing
gitleaks detect --source . --redact
# Example: pre-commit hook idea
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.24.0
hooks:
- id: gitleaks
Scanning is a seatbelt, not a license to be careless. It will miss custom tokens, encrypted blobs, screenshots, and secrets pasted into places outside Git.
Design For Rotation
A secret that cannot be rotated safely is a business risk. Every important secret needs an owner, a purpose, a creation date, a rotation process, and a revocation process. OWASP’s Secrets Management guidance emphasizes lifecycle management: creation, rotation, revocation, and expiration.
Secret inventory fields: Name: Environment: Owner: System: Permissions: Created: Last rotated: Rotation method: Where used: Emergency revocation steps: Business impact if revoked:
For APIs that support multiple active keys, rotate with overlap. Create a new key, deploy it, verify traffic uses it, then revoke the old key. For databases, use separate users per application so rotation does not break unrelated systems.
CI/CD Needs Extra Care
Build systems often hold the most dangerous secrets because they can deploy code, publish packages, access cloud accounts, and read production configuration. Limit which branches and workflows can access secrets. Do not expose production secrets to pull requests from forks. Prefer short-lived cloud credentials through workload identity or OIDC federation instead of long-lived static keys.
# GitHub Actions pattern: prefer OIDC to static cloud keys
permissions:
id-token: write
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure cloud credentials
uses: cloud-provider/oidc-login-action@v1
The exact action depends on your cloud provider, but the principle is the same: issue short-lived credentials to the job that needs them, with the least permission required.
What To Do When A Secret Leaks
Do not debate whether someone used it. Revoke first, investigate second. Assume automated scanners saw the secret if it touched a public repository or public log. After revocation, check usage logs, rotate dependent credentials, remove the secret from history where practical, and document the root cause.
Leaked secret response: 1. Revoke the secret immediately 2. Create a replacement through the approved process 3. Search logs for use before and after exposure 4. Identify data or systems the secret could access 5. Rotate downstream secrets if trust is uncertain 6. Remove the secret from current code and documentation 7. Add a detection rule or pre-commit check 8. Record what changed so the leak does not repeat
A Small-Team Starting Point
If you have no secrets program today, start with four controls: a password manager for human-held secrets, a cloud secret manager for production applications, repository secret scanning, and a monthly review of high-value tokens. That is enough to reduce the most common leaks without slowing the team to a crawl.
Further Reading
Useful official references include the OWASP Secrets Management Cheat Sheet, OWASP guidance on CI/CD secrets, GitHub secret scanning documentation, and your cloud provider’s secret manager documentation.