Armalo Labs
The bench measures cold start, RAM per agent, idle-state size, 1M-agents-per-month hosting cost, and a SWE-bench-equivalent solve rate. Every number cites a source. Unmeasured cells stay unmeasured.
We are not going to publish numbers we cannot reproduce. The competitor vendors publish numbers from their own benches; we run our own bench on the Armalo substrate + the same competitor substrates where we can afford the spend. The benchmark tool is open source and committed to the repo (scripts/runtime-bench/bench.mjs) so the next person who runs it sees the same result.
Once the bench runs, each cell populates with a value + a timestamp + a source citation. The matrix stays empty in this commit because the bench is owned by the infrastructure team, not the marketing team.
Every cell that carries a value was measured by us with a reproducible bench harness (`scripts/runtime-bench/bench.mjs`) or by an external source cited in-line. Unmeasured cells render as — and stay unmeasured until the bench runs.
measured 0 / 48 cells · last updated 2026-08-09T00:00:00Z
Unit: ms · 8 substrates
| substrate | Armalo substrate (E2B + Hetzner + Cloudflare) | Firecracker microVM (self-hosted) | Naïve Vetta (V8 isolate) | E2B hosted sandbox | Modal | Cloudflare Workers | Morph | AWS Fargate |
|---|---|---|---|---|---|---|---|---|
| cold start | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured |
Unit: mb · 8 substrates
| substrate | Armalo substrate (E2B + Hetzner + Cloudflare) | Firecracker microVM (self-hosted) | Naïve Vetta (V8 isolate) | E2B hosted sandbox | Modal | Cloudflare Workers | Morph | AWS Fargate |
|---|---|---|---|---|---|---|---|---|
| ram per agent | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured |
Unit: bytes · 8 substrates
| substrate | Armalo substrate (E2B + Hetzner + Cloudflare) | Firecracker microVM (self-hosted) | Naïve Vetta (V8 isolate) | E2B hosted sandbox | Modal | Cloudflare Workers | Morph | AWS Fargate |
|---|---|---|---|---|---|---|---|---|
| idle state | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured |
Unit: usd · 8 substrates
| substrate | Armalo substrate (E2B + Hetzner + Cloudflare) | Firecracker microVM (self-hosted) | Naïve Vetta (V8 isolate) | E2B hosted sandbox | Modal | Cloudflare Workers | Morph | AWS Fargate |
|---|---|---|---|---|---|---|---|---|
| 1m agents / mo | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured |
Unit: pct · 8 substrates
| substrate | Armalo substrate (E2B + Hetzner + Cloudflare) | Firecracker microVM (self-hosted) | Naïve Vetta (V8 isolate) | E2B hosted sandbox | Modal | Cloudflare Workers | Morph | AWS Fargate |
|---|---|---|---|---|---|---|---|---|
| swe-bench-equiv solve rate | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured |
Unit: usd · 8 substrates
| substrate | Armalo substrate (E2B + Hetzner + Cloudflare) | Firecracker microVM (self-hosted) | Naïve Vetta (V8 isolate) | E2B hosted sandbox | Modal | Cloudflare Workers | Morph | AWS Fargate |
|---|---|---|---|---|---|---|---|---|
| cost per solved | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured | — Unmeasured |
unit helper: ms · usd · pct
The bench lives at scripts/runtime-bench/bench.mjs. It provisions a fresh sandbox on each substrate, takes the cold-start timing from the provider's API, and writes the matrix back to apps/web/components/lab/runtime-benchmark-data.ts. Required env: ARMALO_HETZNER_TOKEN, E2B_API_KEY, and the provider keys you want to compare against.
Treat the matrix as marketing surface, not buyer diligence. The docs/PRD-CURRENT-STATE.md row "Production sandbox substrate" remains the canonical authority until the bench produces a winner.
# Dry-run plan + env check (no measurements)
node scripts/runtime-bench/bench.mjs --dry-run
# Full bench — populates the matrix once the infra lane wires the SDKs
ARMALO_HETZNER_TOKEN=... E2B_API_KEY=... NAIVE_API_KEY=... \
node scripts/runtime-bench/bench.mjs \
--substrate=armalo-substrate,firecracker,e2b