CloudNativePG

Renaming a cluster + its S3 archive

Renames zenml-db-prod → zenml-db-staging, archive prefix zenml-prod/ → zenml-staging/.

Do not aws s3 sync the old archive to the new prefix. The new cluster's archive destination must be empty at bootstrap or the pre-flight check fails with Expected empty archive.

1. Backup prod

The cluster backs up via the barman-cloud plugin, so kubectl cnpg backup fails (cluster has no backup section). Apply a plugin-method Backup instead:

kubectl apply -f - <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: zenml-db-prod-manual-preflip
  namespace: postgres
spec:
  cluster:
    name: zenml-db-prod
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io
EOF

kubectl get backup zenml-db-prod-manual-preflip -n postgres -w   # wait: Completed

2. Provision the new cluster

Ensure the destination is empty:

aws s3 rm --recursive s3://hotosm-k8s-db-backup/zenml-staging/

Add zenml-db-staging-creds sealed secret, then databases/zenml-db-staging.yaml:

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: zenml-db-staging-store
  namespace: postgres
spec:
  retentionPolicy: "60d"
  configuration:
    destinationPath: s3://hotosm-k8s-db-backup/zenml-staging
    endpointURL: https://s3.amazonaws.com
    s3Credentials:
      accessKeyId:     { name: s3-creds, key: access-key-id }
      secretAccessKey: { name: s3-creds, key: secret-access-key }
    wal:  { compression: gzip, encryption: AES256 }
    data: { compression: gzip, encryption: AES256, jobs: 2 }
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: zenml-db-staging
  namespace: postgres
spec:
  instances: 1
  imageName: "ghcr.io/cloudnative-pg/postgresql:18-system-trixie"
  storage:    { storageClass: gp3, size: 20Gi }
  walStorage: { storageClass: gp3, size: 40Gi }
  bootstrap:
    recovery:
      source: zenml-db-prod
  externalClusters:
    - name: zenml-db-prod
      plugin:
        name: barman-cloud.cloudnative-pg.io
        parameters:
          barmanObjectName: zenml-db-prod-store
          serverName: zenml-db-prod
  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters:
        barmanObjectName: zenml-db-staging-store

Commit, let ArgoCD sync.

3. Verify

kubectl get cluster zenml-db-staging -n postgres -w   # In Healthy state
kubectl exec -n postgres zenml-db-staging-1 -- \
  psql -U postgres -d zenml -c "SELECT count(*) FROM pipeline_run;"

4. Switch

Once staging is Healthy and the counts match, point the app at it in apps/zenml/helm/values.yaml:

zenml:
  database:
    url: "postgresql://zenml@zenml-db-staging-rw.postgres.svc.cluster.local:5432/zenml"

Commit, let ArgoCD sync - the URL change rolls the Deployment, which reconnects to the new DB (retrying until it's up). Then add a ScheduledBackup targeting zenml-db-staging.

Writes to prod between step 1 and now won't reach the new cluster - use replica cluster mode if that matters.

5. Drop the old cluster + S3 archive

After one successful scheduled backup on the new cluster:

kubectl delete cluster         zenml-db-prod        -n postgres
kubectl delete objectstore     zenml-db-prod-store  -n postgres
kubectl delete scheduledbackup zenml-db-prod-backup -n postgres
kubectl delete sealedsecret    zenml-db-prod-creds  -n postgres
aws s3 rm --recursive s3://hotosm-k8s-db-backup/zenml-prod/

Remove databases/zenml-prod.yaml and databases/zenml-db-prod-creds.yaml.