Image: Supabase / supabase.com Supabase Just Released an Open Benchmark That Tests AI Coding Agents on Real Backend Tasks
Supabase Evals runs Claude Code, Codex, and OpenCode against actual Supabase tasks — building schemas, fixing Edge Functions, debugging RLS policies — in real environments with real scoring.
Most AI coding benchmarks test whether a model can write a function in isolation. Supabase Evals tests whether an agent can actually build something with Supabase — create a schema, fix a broken Edge Function, debug a Row Level Security policy — in a real environment, not a sandbox mock.
The framework went public under Apache 2.0 on July 31. It runs locally via pnpm, boots a hosted-like Supabase stack alongside the agent under test, and scores output with a combination of deterministic checks and LLM-as-a-judge. The leaderboard lives at supabase.com/evals.
How it works
Every scenario runs against a real environment. The agent gets access to the actual Supabase MCP server and CLI, running in containers. There’s no simulated API or pre-populated mock — if an agent calls a database query, it’s calling a real one.
Scoring mixes hard checks (did the schema actually apply, does the RLS policy enforce correctly) with LLM evaluation for things that are harder to measure automatically.
The three agent frameworks tested at launch were Claude Code, Codex, and OpenCode.
What the results showed
On Build tasks — writing schema, creating functions, standing up features from scratch — Opus 5 and Kimi K3 both scored 100% without any special prompting or Supabase-specific setup. Sonnet 5 came in at 78% unaided but reached 100% when Supabase Skills were loaded. GPT-5.6 Sol improved from 89% to 100% with Skills.
The interesting finding wasn’t that top models can build Supabase apps. It’s what they do wrong when they fail.
Supabase found that agents heavily prefer writing manual SQL migrations over using declarative schema workflows — even when the declarative path is faster. Claude Code checked the Supabase documentation in fewer than 40% of scenarios, suggesting it relies on training knowledge rather than current docs. That gap matters when APIs change or when working with platform-specific features that aren’t well-represented in pretraining data.
Why this is worth paying attention to
Most agent benchmarks are general-purpose: they test coding across a variety of languages and problem types. Supabase Evals is opinionated — it’s specifically about whether agents can build on Supabase correctly.
That’s actually more useful for teams choosing an AI coding tool for backend work. Knowing that Claude Code writes working RLS policies 87% of the time on first try is more actionable than knowing it scores 81 on SWE-bench.
The framework also drives Supabase’s own development. They run it as an internal regression suite to catch cases where agent behavior breaks as their APIs evolve, and to verify that documentation improvements actually help agents complete tasks.
The code is at github.com/supabase/evals and the public leaderboard updates as Supabase runs new results.
Sources: Supabase blog · MarkTechPost · RuntimeWire