The Secrets Problem


Every application depends on secrets: API keys, database passwords, encryption keys, OAuth tokens, and TLS certificates. Mishandling these secrets is one of the most common causes of security breaches. A single hardcoded credential committed to a public repository can compromise your entire infrastructure within minutes.


Where Secrets Go Wrong


| Mistake | Consequence |

|---------|-------------|

| Hardcoded in source code | Credentials exposed in version control |

| Stored in environment files committed to git | Accidental exposure in public repos |

| Shared via chat or email | Unbounded access, no audit trail |

| Stored in config files with wide permissions | Accessible to any process on the machine |

| Logged during debugging | Credentials visible in log aggregation systems |


The Vault Pattern


A secrets vault is a centralized service that stores, manages, and audits access to secrets. Applications request secrets at runtime rather than reading them from configuration files.


HashiCorp Vault



# Start Vault in development mode

vault server -dev



# Store a secret

vault kv put secret/database \

    host=db.example.com \

    port=5432 \

    username=app_user \

    password=$(openssl rand -base64 32)



# Read a secret

vault kv get secret/database

Vault Integration with Application Code



import hvac



client = hvac.Client(url='https://vault.example.com',

                     token=get_vault_token())

secret = client.secrets.kv.read_secret_version(

    path='database'

)

db_password = secret['data']['data']['password']



# Use the secret to connect

conn = psycopg2.connect(

    host=secret['data']['data']['host'],

    password=db_password

)

Cloud-Native Solutions


AWS Secrets Manager



import boto3

from botocore.exceptions import ClientError



def get_secret(secret_name):

    client = boto3.client('secretsmanager')

    response = client.get_secret_value(SecretId=secret_name)

    return json.loads(response['SecretString'])



# Automatic rotation

db_creds = get_secret('prod/database/credentials')

GCP Secret Manager



from google.cloud import secretmanager



def access_secret_version(secret_id, version_id="latest"):

    client = secretmanager.SecretManagerServiceClient()

    name = f"projects/my-project/secrets/{secret_id}/versions/{version_id}"

    response = client.access_secret_version(name=name)

    return response.payload.data.decode("UTF-8")

Secrets in CI/CD


Never store secrets in repository CI/CD configuration files that are committed to source control. Use each platform's native secrets store:


| Platform | How to Store Secrets |

|----------|---------------------|

| GitHub Actions | Settings > Secrets and variables > Actions |

| GitLab CI | Settings > CI/CD > Variables |

| CircleCI | Project Settings > Environment Variables |

| Jenkins | Manage Jenkins > Credentials |



# GitHub Actions workflow with secrets

name: Deploy

on: [push]

jobs:

  deploy:

    runs-on: ubuntu-latest

    steps:

      - uses: actions/checkout@v4

      - name: Deploy to production

        env:

          DB_PASSWORD: ${{ secrets.DB_PASSWORD }}

          API_KEY: ${{ secrets.API_KEY }}

        run: ./deploy.sh

Detecting Secret Leaks


Pre-Commit Hooks


Use tools like `git-secrets` or `truffleHog` as pre-commit hooks:



# Install git-secrets

brew install git-secrets

git secrets --install

git secrets --register-aws



# Scan for secrets before commit

#!/bin/sh

# .git/hooks/pre-commit

git secrets --scan

Scanning Repositories



# Scan the entire history

trufflehog git https://github.com/example/repo.git



# Scan with Gitleaks

gitleaks detect --source . --verbose

Rotation and Expiration


Secrets should have a limited lifetime. Implement automatic rotation:


  • **Database passwords**: Rotate every 90 days. Use zero-downtime rotation with two-password windows.
  • **API keys**: Rotate immediately if compromised, regularly every 180 days.
  • **TLS certificates**: Automate renewal with Let's Encrypt and cert-manager.
  • **SSH keys**: Rotate on employee departure or annually.

    • 
# Example: Zero-downtime DB password rotation

# Phase 1: Update app to accept both old and new passwords

# Phase 2: Rotate master password in database

# Phase 3: Update app to use only new password

# Phase 4: Revoke old password

    The Principle of Least Privilege


    Secrets should be scoped to what a particular service or developer needs:


  • Each microservice gets its own set of secrets, not shared credentials.
  • Developers get ephemeral secrets scoped to development environments.
  • Service accounts have the minimum permissions required for their function.
  • Audit every secret access with timestamp and requester identity.

    • Summary


    Treat secrets as the critical infrastructure they are. Use a dedicated secrets vault in production, store them in platform-native secret stores for CI/CD, never hardcode them or commit them to version control, and implement automatic rotation. Combine these practices with regular scanning for leaked secrets and strict access control to minimize the blast radius of any exposure.