Skill issue: Lessons from skilling up coding agents to use Langfuse - Marc Klingen, Clickhouse
Summary
Marc Klingen, co-founder of Langfuse — the largest open-source LLM observability and evaluation platform by self-reported metrics — presents six concrete learnings from building a Claude Code skill to help developers instrument their applications with Langfuse. The talk spans approximately 24 minutes and is grounded in the real engineering challenges of scaling developer onboarding from human-assisted support calls to autonomous agent-guided setup.
Klingen opens by framing skills as a formalized shortcut that bridges the reliability of deterministic workflows with the flexibility of fully autonomous agents. He uses a Rubik's Cube analogy: just as a manual transforms a confusing puzzle into a solvable sequence, a skill gives an agent the structured context it needs to perform reliably. The core problem motivating the skill was Langfuse's 478-page documentation spanning five feature areas (tracing, evals, prompt management, experiments, and a CLI), combined with a highly flexible, non-opinionated architecture. Without a skill, Claude Code would implement instrumentation using stale pre-training context, ship broken code, detect the failure, then fetch updated docs to self-correct — an inefficient two-pass process that produced traces capturing only raw LLM calls with no visibility into actual agent behavior.
The six learnings Klingen details are: (1) Studying runtime traces directly provides 80% of the signal needed to improve a skill. Klingen's team instrumented Claude Code itself via Langfuse, watched execution traces, and identified specific failure modes such as hallucinated CLI parameters (the agent inferred plausible-sounding flags from the word 'trace'). The fix was to prominently advertise the --help flag so the agent always inspects the actual CLI capabilities. (2) Helping agents navigate information rather than crawl it — Langfuse exposed an agent-focused sitemap and enforced markdown content negotiation (appending .md to any docs URL) to avoid agents parsing HTML and wasting tokens. (3) Surfacing a natural-language RAG search endpoint over documentation rather than forcing agents to fetch individual pages. This endpoint, rebuilt from Langfuse's existing docs Q&A stack, also captures search queries — enabling the team to identify undocumented problem areas from real agent usage patterns. (4) Implementing even a basic eval setup is better than none. Langfuse created five reference evaluation templates paired with LLM-as-a-judge checks against filesystem diffs (before/after running the skill) to catch regressions, such as missing retrieval spans in RAG traces. (5) Dynamic documentation content should be referenced by URL, not duplicated inside the skill, to prevent the skill from becoming a stale secondary cache of information — the same problem that afflicts pre-training context. (6) Auto-research loops with target functions can accelerate skill improvement but are tightly bounded by how well the target function is defined. Langfuse ran an automated research loop targeting prompt migration from local Git repositories into Langfuse Prompt Management, with a target function initially measuring speed (number of turns). The agent optimizing for fewer turns removed documentation-fetching steps — the very steps that keep the skill current — nearly defeating the purpose of the skill. Three of six suggested improvements were accepted after human review.
Klingen closes with open challenges: skill versioning and distribution (no package management exists; his team is exploring timestamp-based staleness alerts), the tension between fast initial onboarding versus deep, correct setups, and the question of whether skills should require explicit user installation or be auto-discoverable. He frames the roadmap as moving toward a fully automated evaluation lifecycle — connecting repositories to Langfuse and letting an agent continuously run evals, surface prompt version impact on production results, and synthesize user feedback patterns — without manual UI interaction.
Key Takeaways
- •Analyze runtime execution traces of your coding agent before modifying a skill — this alone provides roughly 80% of the signal needed to diagnose and fix skill failures.
- •Expose a natural-language search endpoint over your documentation corpus so agents can retrieve relevant chunks in one call instead of crawling hundreds of pages sequentially.
- •Enforce markdown content negotiation in your skill (e.g., append .md to docs URLs) to prevent agents from parsing HTML and wasting tokens on unnecessary content.
- •Publish an agent-focused sitemap so the agent knows what documentation exists and can navigate directly to the right section rather than relying on general web search.
- •Reference documentation by URL inside the skill rather than duplicating content — duplication creates a second stale cache with the same pre-training-context problem you're trying to solve.
- •Define a minimal but representative eval suite (even 5 templates) with LLM-as-a-judge checks against before/after diffs — something is far better than nothing for catching regressions.
- •When using auto-research loops to improve a skill, define the target function carefully: optimizing for speed (fewer turns) caused the agent to strip documentation-fetching steps, breaking the skill's ability to stay current.
- •Human review of auto-research suggestions is essential — Langfuse accepted only 3 of 6 proposed improvements because their target function was imperfect.
- •Prominently advertise CLI --help flags in your skill to prevent agents from hallucinating plausible-sounding but non-existent command parameters.
- •Track search queries made to your documentation endpoint to identify undocumented problem areas based on real agent and user behavior.
Transcript
Okay, that was great. Hi everyone, super excited to be here. I'm Mark, one of the founders of LangFuse. We started LangFuse like 3 years ago when everything was felt quite early. Building agents that didn't work and then realized, "Okay, there needs to be like some evaluation tracing." Built like LangFuse is the open-source project in the space by now. And with the metrics that we track, we seem to be the largest one in the space. We do our product engineering out of Europe. Thus, I'm very excit...
Speakers
Topics
Related Sessions
