Skip to content

S3 Credential Management for Argo Workflows

Context and Problem Statement

Each ODM (OpenDroneMap) job in ScaleODM needs to download raw imagery from S3 and upload processed outputs back to S3. The system uses Argo Workflows for long-running processing jobs, often lasting hours or days.

Considered Options

  • Use long-lived static AWS credentials in Kubernetes Secrets.
  • Use short-lived AWS STS credentials per workflow.
  • Use pre-signed URLs for each job's S3 read/write operations.

Decision Outcome

We chose to use static S3 credentials stored in a Kubernetes Secret, referenced by workflow pods via secretKeyRef. This was chosen over STS temporary credentials because:

  • ODM jobs can run for many hours and may be queued for extended periods before starting. STS session tokens (max 12h) can expire before a queued job begins processing, causing silent failures.
  • The attack surface is small: credentials are only accessible within the Kubernetes cluster via the Secret API, never inlined in Argo Workflow specs.
  • Operational simplicity: no IAM role trust policies, no STS endpoint configuration, no per-job credential generation.

How it works

  1. The operator creates a Kubernetes Secret (scaleodm-secrets) in the ScaleODM release namespace containing AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION. In the unified runtime secret contract this same secret also contains SCALEODM_DATABASE_URL and AWS_S3_ENDPOINT.
  2. ScaleODM builds Argo Workflow specs that reference this secret via secretKeyRef env vars - credentials are resolved at pod startup, not baked into the workflow manifest.
  3. All workflow containers (download, process, upload, cleanup) use rclone with env_auth = true to pick up credentials from the environment.

Example Argo workflow env vars

env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: scaleodm-secrets
        key: AWS_ACCESS_KEY_ID
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: scaleodm-secrets
        key: AWS_SECRET_ACCESS_KEY
  - name: AWS_DEFAULT_REGION
    valueFrom:
      secretKeyRef:
        name: scaleodm-secrets
        key: AWS_DEFAULT_REGION
        optional: true

Consequences

  • Credentials are never visible in kubectl get workflow -o yaml.
  • No expiry concerns for long-running or queued jobs.
  • Single credential pair for both read and write operations (same IAM user/role needs s3:GetObject and s3:PutObject on the relevant buckets).
  • Credential rotation requires updating the Kubernetes Secret and restarting affected pods.