Skip to content

ML pipeline

The backend never trains or runs inference inline. It submits pipelines to ZenML and polls their state. The choice of ZenML as the orchestrator is recorded in the MLOps decision record. The services behind the pipeline (STAC, MinIO, MLflow, and ZenML) and their roles are listed in the Architecture overview.

Deployment architecture

These services run together on Kubernetes. The fAIr backend submits pipelines to ZenML and manages ONNX serving through Knative. STAC is the source of truth for dataset and model metadata, S3 stores artifacts, MLflow tracks experiments, and Postgres stores service state. How HOT runs this on open source over AWS is described in How HOT uses open source on AWS to power humanitarian AI.

fAIr deployment architecture: developers push to fAIr-models on GitHub, CI/CD builds the model image to the registry and registers base models in the STAC model registry; the fAIr backend submits jobs to ZenML, which orchestrates training on autoscaling GPU nodes and ONNX inference through Knative on CPU nodes, with S3 as the artifact store, MLflow as the experiment tracker, and Postgres for state.

flowchart LR
    M[fAIr-models] -->|CI builds images| R[GHCR]
    A[Admin] -->|register pinned item| B[fAIr backend]
    R --> Z
    R --> K
    B --> S[(STAC)]
    B --> O[(S3 artifacts)]
    B --> P[(Postgres)]
    B --> K[Knative serving]
    B --> Z[ZenML]
    Z --> G[Autoscaling GPU jobs]
    Z --> O
    Z --> F[MLflow]
    K --> C[CPU inference]

Contribute and register a model

A model developer contributes through the fAIr-models repository; see Contributing a model. CI validates the pull request and, after merge, publishes its training and inference images. An admin pins both image references to digests once and registers the same item in staging and production. Registration mirrors the weights into object storage and publishes the item to the environment's base-models collection.

flowchart TD
    A[Model developer] -->|Prepares PR| B[fAIr-models GitHub]
    B -->|CI: build, validate, test| C{Review}
    C -->|Merge| D[Publish training + inference images]
    D --> E[Admin pins image digests once]
    E -->|same item| F[Register and test in staging]
    F -->|approve| G[Register in production]
    G --> H[Production STAC + live Knative route]

Inside fAIr, registration is admin-only and asynchronous. What it does, step by step, is in Register a base model.

Local compose has no Knative

The inference-service deployment applies to the production Kubernetes setup. A local docker compose stack has no Knative, so that step is skipped.

The contribution diagram is adapted from the fAIr-models architecture docs.

Training

POST /trainings/submit/ starts training asynchronously (see Key invariants). A worker submits a ZenML pipeline that runs split, train, eval, and onnx, and polls status into the database. The run produces weights.pt and model.onnx in MinIO. A base model with no mlm:training asset is prediction-only, and submitting a training for it returns 400.

POST /trainings/{id}/publish/ writes the versioned local-models/ STAC item.

flowchart LR
    A[submit] --> B[worker submits<br/>ZenML pipeline]
    B --> C[split -> train -> eval -> onnx]
    C --> D[weights.pt + model.onnx<br/>in MinIO]
    D --> E[publish -> versioned<br/>local-model STAC item]

Prediction

POST /predictions/submit/ enqueues inference. A worker downloads chips for the requested bbox, submits an inference pipeline, and a post-run step turns the GeoJSON into .fgb and .pmtiles, after which GET /predictions/{id}/result/ returns three presigned URLs. A model with neither an mlm:training nor a batch-inference asset serves try-now only, and submitting a prediction for it returns 400.

flowchart LR
    A[submit] --> B[worker downloads chips,<br/>runs inference]
    B --> C[post-run:<br/>geojson -> fgb + pmtiles]
    C --> D[results_ready = true]
    D --> E[GET result/<br/>geojson, fgb, pmtiles]