Skill Issue: How We Used AI to Make Agents Actually Good at Supabase — Pedro Rodrigues, Supabase

Pedro Rodrigues, AI Tooling Engineer at Supabase, delivered a hands-on workshop at AI Engineer Europe 2026 focused on writing, testing, and iterating on Agent Skills to genuinely improve agent performance in real-world Supabase workflows. The session covered the full lifecycle of skills development — from writing a skill.md file through manual testing to automated evaluation pipelines using Braintrust.

Rodrigues opened by distinguishing skills from MCP tools. Skills are folders containing a skill.md file (with a YAML front matter name and description, plus optional reference markdown files and scripts) that leverage progressive disclosure — only the front matter is loaded to context initially, and the full content is fetched on demand when the agent determines it is relevant. This contrasts with MCP tools, which load all tool definitions upfront. He argued that skills and MCP are complementary: MCP is ideal for integrations and remote tool execution that doesn't depend on the local environment, while skills are best for providing rich contextual instructions, workflow guidance, and scripts that run in the agent's local environment.

The practical demonstration centered on a vibe-coded Next.js performance-review application backed by Supabase. Without any skill loaded, Rodrigues prompted Claude Code to create a SQL view (department_stats) showing headcount and average salary per department. Claude successfully created the view and declared success — but the view bypassed Postgres row-level security (RLS). The root cause: in Postgres, views created without the SECURITY INVOKER flag execute with the permissions of the view creator, not the calling user, silently bypassing any RLS policies on the underlying tables. Employees who should see only their own data could see the entire company's salary data.

Rodrigues then installed a pre-prepared Supabase Security skill using the Vercel `skills` NPX package, which installs skills into a `.agents/` folder and creates a symlink to `.claude/skills/` so Claude Code can find them. The skill's description was deliberately prefixed with the verb "use" — an empirical finding from his experiments that significantly increases the probability of the skill being loaded organically by Claude, compared to other description phrasings. With the skill active, re-running the same prompt caused Claude to include the `WITH (security_invoker = true)` flag when creating the view, demonstrating how a targeted, human-readable skill.md can reliably alter agent behavior.

The second half of the workshop introduced an evaluation-driven development loop based on the Agent Skills Open Standard (AgentSkills.org) framework. Rodrigues described three grading mechanisms: deterministic tool-call assertions (checking whether specific MCP tools were called), expected output string checks, and LLM-as-a-judge for non-deterministic output evaluation. He showed an eval.json structure with prompt, expected_output, and assertions fields. The Python eval harness resets the Supabase database state via `npx supabase db reset` before each run, then runs Claude Code in headless/print mode as a binary, once with the skill injected into the system prompt and once without, comparing grading.json outputs across both conditions.

A live evaluation run surfaced an important meta-lesson: the assertions themselves were misconfigured — Rodrigues had written a check that inspected `pg_class` RLS options rather than directly querying the view definition, causing the harness to report the with-skill condition as failing even though the view was correctly created. He used this failure to illustrate the hardest part of eval-driven development: defining representative, correctly-implemented test scenarios. He noted that non-deterministic LLM-as-a-judge graders can also hallucinate, compounding the challenge.

Rodrigues closed with production advice: treat skills as living documentation requiring the same maintenance discipline as code — update them when workflows change, periodically verify they are still being loaded by users, and prune unused skills from CI environments while keeping local experimentation unconstrained. He also flagged that Anthropic's tool-search mechanism (progressive disclosure for MCP tools) is currently Claude Code-specific and not yet part of the MCP protocol, but noted the MCP co-founder is expected to announce protocol-level progressive disclosure for tools at the conference.