Coding agents spent 30 hours flashing firmware onto real hardware, and found a bug in Matter
GroundRun gave Claude Code agents a hardware rig and one job: implement six OTA update methods for Matter over Thread on an ESP32-H2 and measure them. All thirty validation runs passed.
The newest coding-agent benchmark does not run in a sandbox. It runs on a desk. In an October 5 post, the team at GroundRun describes giving AI coding agents a hardware rig and a single goal: implement over-the-air update mechanisms for a Matter smart-home device running on an ESP32-H2 chip, validate them on real hardware, and measure the results.
The rig was an ESP32-H2 devboard and a Silicon Labs EFR32MG24 devboard acting as a Thread border router, plus an Android phone, all wired to a Raspberry Pi-class GroundRun Desk runner. The coding agent was Claude Code running Claude Opus 5.5 at medium effort. The agents worked for 29.7 hours, mostly unattended, through a pipeline of survey, planning, and execution agents that produced twelve tasks.
The assignment: implement six ways to deliver a firmware update to the board over Matter's Thread network, then run each one five times. The variants covered a full 1.6 MB image download at 1,024, 512, and 256-byte block sizes; a delta update carrying only the changed bits; an encrypted image decrypted block by block on the device; and, as a baseline, a Bluetooth update pushed from Espressif's phone app.
The numbers are refreshingly concrete. A full image at the default 1,024-byte blocks took a median 8 minutes 38 seconds to transfer, with the board back on the Thread network at 8:47. Smaller blocks were slower, 12:16 at 512 bytes and 21:22 at 256, thanks to the stop-and-wait block protocol. Encryption cost nothing measurable: 8:33 against the plain 8:38, with overlapping ranges. The Bluetooth baseline came in at 8:04. The delta update, a 61 KB patch against the 1.6 MB image, transferred in 32 seconds, with the board back on Thread in 41.
Every one of the thirty runs succeeded. Along the way the agents tripped over a genuine bug in the Matter codebase while implementing the delta variant, then moved all six variants to a newer Matter release that fixed it and re-ran everything.
GroundRun's framing is that OTA deserves this level of rigor because it is the mechanism that keeps connected devices secure in the field, and a failed update bricks the device. The EU's Cyber Resilience Act now mandates security updates for connected products, which makes mechanically verified update paths a compliance question rather than just an engineering one.
The honest limits: one chip, one model, one lab's rig. But as an existence proof it is hard to beat. Coding agents planned the work, wrote the firmware variants, drove the hardware, measured the results, and caught a real bug in an upstream codebase, with humans mostly watching. Benchmarks that run on actual silicon are still rare. Expect more of them.
