Can LLMs generate Enterprise Quality Code? — Prasenjit Sarkar, Sonar

Prasenjit Sarkar from Sonar presents findings from a large-scale empirical study evaluating whether LLMs generate enterprise-quality code, drawing on data from 4,444 distinct Java programming assignments run through 53 models (and their variants). The study was motivated by the rapid shift to agentic development — citing a Pragmatic Engineer Survey from March 2026 showing that 55% of developers now regularly use AI agents — and questions whether the code those agents produce is maintainable, secure, and readable.

Sarkar argues that standard LLM leaderboards (HumanEval, MBPP, SWE-bench) measure only functional correctness (pass rate) and miss critical engineering dimensions: security vulnerabilities, bug density, code maintainability, and cyclomatic and cognitive complexity. Sonar's own evaluation framework ran all 4,444 assignments through SonarQube Enterprise and produced a publicly accessible leaderboard at sonar.com/leaderboard.

Key benchmark findings across models: - GPT-4o: under 250,000 lines of code for the full assignment set (relatively concise) - Claude Sonnet 4.6: 627,000 lines of code; highest security issue rate at 300 security issues per million lines of code - GPT-5.4 and GPT-5.4 Pro High: 1.2 million lines of code — the most verbose by a wide margin - Gemini 3.1 Pro High: highest pass rate at 84.17%; 307,000 lines of code; 614 bugs per million lines; 210 security issues per million lines - Five models exceed 80% functional correctness, but all exhibit meaningful code-quality problems

A notable trend: newer models generate progressively more lines of code ("verbosity going north"), and while reinforcement learning is reducing known security flaw categories, it introduces subtler, harder-to-detect bugs in their place. Root causes identified include mixed-quality training data, security flaws embedded in open-source training corpora, hidden logic errors in training sets, and LLMs' probabilistic nature meaning the same prompt yields different outputs on each run.

Sarkar also distinguishes cyclomatic complexity (branch count: if/else, for, while loops) from cognitive complexity, a Sonar-proprietary metric measuring how difficult code is for a human to read and maintain — a nuance absent from most LLM evaluations.

Sonar's response is the ACDC framework (agent-centric development cycle), a three-stage pipeline: Guide, Verify, Solve. In the Guide phase, two products address upstream data quality: Sonar Sweep (private beta) cleans training data before it reaches the model, and Context Augmentation pushes the full company codebase into the LLM's context window. In the Verify phase, SonarQube Agentic Analysis (open beta) integrates via MCP into tools like Claude, Codex, and Gemini CLI, running static analysis in 1–5 seconds before a commit — compared to 1–5 minutes in CI — catching issues before they reach a pull request. In the Solve phase, the SonarQube Remediation Agent (open beta) handles issues that slip through to PRs: it creates one fix per issue, runs it through both static analysis and compilation to check for regressions, and only presents the fix to the developer if it passes both checks. The remediation agent can also address existing tech debt in bulk — developers select issues from the SonarQube dashboard, assign them to the agent, and receive one PR per issue for human review. The framework supports 40+ programming languages and integrates with major DevOps toolchains and IDEs.