AI × Embedded Firmware

Embedded Vibecode Bench

Tracking how AI models do at embedded firmware. Each model gets a frozen one-shot prompt and must write a complete bare-metal firmware from scratch: linker script, startup, drivers, build script, everything. It's compiled and run on Renode simulations of real Cortex-M boards and scored objectively from diagnostics (compiles → boots → behavioral checks). Every cell is run 3× and averaged; same prompts every run, so progress is comparable over time.

Leaderboard

0–100. Reference firmware = 100. A score is how close to known-good on these tasks (see methodology below).

#ModelProviderScore Frontier TasksBy platformReleased

Scores chart

Capability over time
Run-to-run consistency
Value vs time
Value vs cost

Model × task matrix

One cell per model per task, colored by score. Click any cell → opens the full prompt, model output, build log, and Renode trace.

About this benchmark

What it doesn't measure
  • Simulator, not silicon. Renode, not your board. No errata, analog, EMC, or real timing.
  • Bring-up & classic algorithms. CRC, COBS, ring buffers. Not novel architecture or maintainability.
  • Not robustness. No fault injection, brown-out, or security-hardening depth. Don't ship a bootloader on a score.
  • Small n. 3 runs, averaged. Treat small gaps as noise; open a cell for spread.
  • Cortex-M only, for now. ST/Nordic is what Renode runs well. ESP32 & RISC-V are roadmap.
  • Our tasks, our checks. Objective and fully auditable, but read the checks, not just the ranking.