Skip to content

Swap-based Memory Sizing for ODM Jobs

Context and Problem Statement

ScaleODM sizes each ODM process pod from the image count, using a fitted table in app/workflows/standard.go. That table predicts the peak working set: a brief spike (notably mvs-texturing) that lasts seconds to minutes, not the steady-state resident set.

Previously the pod requested the whole peak as real RAM (request ~= limit, Guaranteed QoS). This over-provisioned nodes badly. Field data:

  • 3000 images peaked at ~144Gi but ran fine on a 62Gi RAM + 127Gi swap host.
  • 5000 images ran on ~62Gi RAM + swap, versus a ~227Gi estimated peak.

Provisioning the full peak as RAM means paying for 3-4x the RAM actually resident for all but a few seconds of the job.

Considered Options

  • Request the full peak as RAM (status quo). Simple and safe, but wastes large amounts of RAM and caps how many images a given machine can process.
  • Request steady-state RAM, absorb the peak with swap. Request a fraction of the peak and let the kernel spill the brief spike to disk swap under Kubernetes' LimitedSwap policy.
  • Lower the estimates outright. Rejected: the peak is real and would OOM-kill without swap to catch it.

Decision Outcome

We request the steady-state RAM and let swap catch the brief peak:

request (real RAM) = peak / (1 + SWAP_RATIO)             # prod ratio 1.0, so ~1/2 of peak
limit (swap on)    = request * (1 + MARGIN_PERCENT/100)  # a resident cap below node RAM
limit (swap off)   = peak * (1 + MARGIN_PERCENT/100)     # whole peak as RAM

The limit is the container's memory.max, which caps resident (non-swapped) memory. With swap on it must sit below node RAM (Karpenter sizes the node off the request), so the kernel is forced to push the peak's overflow into swap and the node never fills up. Because request < limit the pod is Burstable QoS and gets a swap grant - Guaranteed pods (request == limit) get none. Nodes provide swap (AWS via the Karpenter EC2NodeClass, other clusters via a disk swapfile).

This lets 5000 images request ~114Gi RAM (peak 227 / 2) + swap instead of a ~227Gi RAM node.

How it works

Swap is split into two independent layers so the same build runs anywhere:

  • Node layer: swap exists on the node and kubelet allows it (failSwapOn: false, NodeSwap feature gate, swapBehavior: LimitedSwap). The node also sets vm.swappiness=10: swap is only needed for the brief peak, so low swappiness keeps ODM's hot working set in RAM during the long steady-state stages and only spills at the peak. Note swappiness biases reclaim toward or away from anon pages even under memory.max pressure, so 10 is a benchmark vs the 60 default. Provisioned per-platform.
  • Workload layer: the pod declares request < limit. Platform-agnostic; ScaleODM never needs to know how a node got its swap.

Each workflow is annotated with scaleodm.hotosm.org/burst-headroom-gib (limit - request) as a rough guide. The real per-pod swap allowance is container_swap_limit_bytes (LimitedSwap grants it proportionally).

Setup and tuning knobs are in docs/swap.md.

Why it stays safe

The container limit is a resident cap that targets below node RAM. When resident hits it, memcg reclaim pushes the overflow to swap, so the node doesn't fill up. This matters because kubelet's node-pressure eviction is swap-blind - its memory.available ignores swap, so free swap alone can't stop it. The old peak-based limit (~2x node RAM) gave the pod no memcg reclaim boundary at all; resident climbed into node RAM and eviction fired even with swap free. (Why node-level global reclaim didn't save it is unproven - the failed run lacked telemetry.)

Backing it up:

  • Swap = 2x node RAM on fast NVMe - plenty to hold the overflow, quick to page.
  • LimitedSwap grants swap proportionally: pod swap = (request / node RAM) * node swap. So node RAM must stay close to the request, or the pod gets too little swap. Selection is driven by RAM (not CPU) to keep that ratio high.
  • Node reclaim tuning (vm.min_free_kbytes, vm.watermark_scale_factor) makes kswapd start early, so a fast fusion spike can't outrun reclaim into eviction.

Containment vs completion. Containment: when the limit lands below node RAM, a node-level eviction (and NotReady risk) becomes a contained, single-pod event. This is not guaranteed at every node size - ScaleODM doesn't know which node Karpenter picks, so a request just under a node's allocatable leaves no room for the margin and the limit can land above node RAM (e.g. request 113Gi on a 128Gi node -> limit 136Gi). Node reclaim tuning is the backstop there; watch such jobs. Completion: not guaranteed either - if too much of fusion's working set is active (not swappable), the limit turns the eviction into a contained cgroup OOM. Field data suggests it's mostly swappable (the ~144Gi peak on a 62Gi RAM + swap host above), but validate before relying on it: verify memory.swap.max is roughly the expected proportional grant (not just > 0), then watch one real large job - swap climbs, memory.events shows no OOM, node stays above 100Mi - before trusting it. If it OOMs, levers in order: lower SWAP_RATIO (bigger request = more RAM + bigger swap share), then more node swap.

Consequences

  • Nodes are provisioned for the resident set, not the peak, cutting RAM cost and raising the images-per-machine ceiling substantially.
  • The estimate extrapolates beyond the 5000-image table (last-segment slope, ~0.043 GiB/image) instead of clamping, so large jobs are sized realistically. MEMORY_MAX_GIB bounds it to what the pool can schedule. At ratio 1.0 (request = peak/2) the ceiling for a 256Gi r-family NVMe node (e.g. r6id.8xlarge
  • 2x swap) is ~8-9k images (peak ~380Gi, ~190Gi RAM request). Beyond that needs bigger-RAM instances, out of scope for this pool.
  • Nodes must provide swap. Without it, a Burstable pod that exceeds its request is OOM-killed at the peak. Enforced on AWS by the swap-enabled EC2NodeClass; on other clusters it is an operator responsibility (docs/swap.md).
  • On AWS, ScaleODM runs on a dedicated tainted pool (pool=scaleodm) so the NVMe/swap nodes are isolated from other workloads and one big job can't starve the shared pool's capacity. See docs/swap.md.
  • Peak-time performance degrades while swapping (disk is slower than RAM), but on AWS the swap is on local NVMe instance store (*id families), ~100x faster than EBS, so heavy paging stays acceptable.
  • Effective OOM headroom depends on the proportional swap formula above, so SWAP_RATIO should be validated per real workload rather than trusted blindly.
  • SWAP_RATIO=0 restores the old behaviour (request the full peak as RAM) for clusters without swap.
  • A job that gets OOM-killed mid-run but reports success is not necessarily fine: OpenMVS surviving an OOM may mean a subprocess or optional stage was skipped. Validation must check expected outputs (orthophoto, DSM, point cloud) and kernel OOM counters (dmesg / node_vmstat_oom_kill), not just the workflow's final status.

CPU sizing

Node selection is driven by RAM. CPU is requested per GiB of the RAM request at the r-family ratio (CPU_PER_GIB=0.125, 1 vCPU per 8Gi), so the CPU request always fits the RAM-selected r-node and never forces a bigger instance.

In production the CPU limit is omitted (CPU_LIMIT_MULTIPLIER=0), so ODM bursts to every core on the node (ODM defaults its concurrency to all detected CPUs, and we pass no --max-concurrency flag). Where a limit is set, ScaleODM passes a --max-concurrency matching it so ODM does not oversubscribe the quota. The generic chart default keeps CPU_LIMIT_MULTIPLIER=1.5 (a bounded limit) as the safe out-of-the-box value; set it to 0 to allow bursting. CPU therefore affects only speed, never feasibility, and a --split submodel workflow is not needed for scheduling (it remains a possible future wall-clock optimisation).

Spot vs on-demand

Jobs at or above ONDEMAND_IMAGE_THRESHOLD (5000) are auto-upgraded from spot to on-demand, because they run for hours and a spot eviction would waste the whole run. Smaller jobs stay on cheap spot. An explicit on-demand request is always honoured.