Skip to content

Architecture

ScaleODM is a single Go service. It takes NodeODM-style API requests and runs each task as an Argo Workflow on Kubernetes. Imagery is read from S3 and results are written back to S3.

flowchart LR
    C[Client<br/>e.g. DroneTM, pyodm] -->|NodeODM API| API[ScaleODM API]
    API --> DB[(Postgres<br/>job metadata)]
    API -->|submit workflow| Argo[Argo Workflows]
    R[Reconciler] -->|poll status| Argo
    R --> DB
    R -->|webhook| C
    Argo --> Pod[ODM job pods]
    Pod -->|read images / write outputs| S3[(S3)]

Lifecycle of a task

  1. Submit. The client calls POST /task/new with readS3Path, writeS3Path and ODM options. The API checks the input, counts the images in S3 and works out how much CPU, memory and disk the job needs.
  2. Run. The API submits an Argo Workflow and records the job in Postgres. The workflow has three steps:
  3. download: rclone copies the images from S3 into a workspace volume.
  4. process: the ODM container runs on the images.
  5. upload: rclone copies the outputs to writeS3Path.
  6. Track. The reconciler is the only writer of job status. Every 30 seconds it saves Argo's phase for each unfinished job, and fails jobs whose workflow has gone missing. GET /task/{uuid}/info and the UI only read: they show the live Argo phase until the job finishes, then the saved status. Pod errors such as an image pull failure show as failed while they last, but are not saved, because Argo can still recover.
  7. Notify. When a job finishes or fails, the reconciler calls the client's webhook, if one was given.

There is no queue of its own. Argo and the Kubernetes scheduler (with Karpenter on AWS) decide when and where pods run.

Code layout

Path Purpose
main.go Startup: config, database, Argo client, reconciler, HTTP server.
app/api/ NodeODM-compatible HTTP API (/task/*, /info, health checks).
app/workflows/ Builds Argo Workflow specs and sizes resources by image count.
app/workflows/scripts/ Shell scripts run by each workflow step.
app/reconciler/ Background loop that syncs Argo status to the database and calls webhooks.
app/meta/ Job metadata store (reads and writes to Postgres).
app/db/ Database connection and schema.sql.
app/s3/ S3 helpers: listing images, output files, download filters.
app/config/ Environment variable config and defaults.
app/ui/ Small web UI at /ui for browsing jobs (read-only by default).
chart/ Helm chart for ScaleODM, with optional Argo and Postgres.

Further reading