Secrets Management In Kubernetes

From skillshare session 26/11/2025

Video Recording

How do secrets work in Kubernetes?

• Kubernetes stores secrets as plain text = compromised cluster = compromised secrets.
• This is generally fine though: all depends on your threat model.
• SealedSecrets facilitate a GitOps approach and keep secrets outside of password managers, and inside Git repos - fully encrypted.

Install Bitnami Sealed Secrets

1. Deploy using ArgoCD:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
spec:
  project: default
  source:
    chart: sealed-secrets
    repoURL: https://bitnami-labs.github.io/sealed-secrets
    targetRevision: 2.18.0
    # We override keyrenewperiod this to provide a simpler GitOps experience.
    #
    # With key renewal enabled - best practice - this means the master key is
    # renewed every 30 days.
    # 
    # In the event of recovery, we need to have made sure we had the latest key
    # backed up - this is unrealistic.
    #
    # Options to backup via cron into object storage are undesirable.
    # Simply keeping the primary key very safe is a simple solution.
    # A tradeoff between operational complexity and security.
    helm:
      releaseName: sealed-secrets-controller
      valuesObject:
        fullnameOverride: sealed-secrets-controller
        keyrenewperiod: 0
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true

1. Install kubeseal:

• Present in ghcr.io/spwoodcock/awscli-kubectl:latest
• Or install manually: https://github.com/bitnami-labs/sealed-secrets/releases

1. Backup Sealed Secrets Key:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >main.key

Store this securely!

Create Sealed Secrets

Using --from-literal (simple values)

kubectl create secret generic cloudflare-api \
    --from-literal=ACCOUNT_ID=your-account-id \
    --from-literal=TOKEN=your-token \
    --dry-run=client \
    --namespace='gateway' \
    -o yaml > secret.yaml

Using --from-file (file contents)

# Single file
kubectl create secret generic my-secret \
    --from-file=config.json \
    --dry-run=client -o yaml > secret.yaml

# Multiple files
kubectl create secret generic tls-secret \
    --from-file=tls.crt \
    --from-file=tls.key \
    --dry-run=client -o yaml > secret.yaml

# Custom key name
kubectl create secret generic my-secret \
    --from-file=my-config=config.json \
    --dry-run=client -o yaml > secret.yaml

Seal and commit

kubeseal -f secret.yaml -w sealed-secret.yaml
rm secret.yaml
git add sealed-secret.yaml

Update Existing Sealed Secret

1. Get current secret from cluster:

kubectl get secret cloudflare-api -n gateway -o yaml

1. View current values:

kubectl get secret cloudflare-api -n gateway -o jsonpath='{.data.TOKEN}' | base64 -d

1. Recreate with new values:

kubectl create secret generic cloudflare-api \
    --from-literal=ACCOUNT_ID=new-account-id \
    --from-literal=TOKEN=new-token \
    --dry-run=client \
    --namespace='gateway' \
    -o yaml > secret.yaml

1. Re-seal and commit:

kubeseal -f secret.yaml -w sealed-secret.yaml
rm secret.yaml
git add sealed-secret.yaml
git commit -m "Update sealed secret"

Migrating a Sealed Secret to a New Namespace

Sealed secrets are bound to namespace + name by default (strict scope), so you can't just edit the metadata.namespace field - the controller will refuse to decrypt. The move requires:

1. Unseal the existing secret to plaintext with the recovery key.
2. Edit the namespace (and anything else) in the plaintext.
3. Reseal with kubeseal targeting the new namespace.

Emergency Recovery

To retrieve a secret from a sealed secret file (offline):

• Get the saved master key.
• Then run:

kubeseal \
  --recovery-unseal \
  --recovery-private-key master-key.key \
  -f sealed-secret.yaml -o yaml > secret.yaml

• This will produce a normal Kubernetes secret, which you can base64 decode as needed.

Other Options

• Cloud provider integrations (AWS Secrets Manager, etc.) - not portable
• External Secrets Operator with Hashicorp Vault - requires external service
• SealedSecrets = simple, portable, GitOps-friendly