Does GenAI "belong" to data scientists? — Phil Hetzel, Braintrust

Phil Hetzel, head of solutions engineering at Braintrust — an agent quality platform focused on evals and observability — argues that generative AI and agentic development should not be siloed within traditional ML or data science teams. Drawing on a year of observations across enterprises and AI-native companies building agents in production, he identifies a systemic organizational mistake: because generative AI has 'AI' in the name, it gets handed to whoever already runs the ML platform, even though the skill profile required is fundamentally different.

Hetzel distinguishes two organizational archetypes. Traditional enterprises delegate GenAI downward from C-suite directives to existing ML/data science platform teams, bringing heavy process tooling but narrow evaluation instincts. AI-native companies, by contrast, start with cross-functional small teams where every engineer is close to the problem and the product surface area simultaneously.

The core technical argument: foundation model providers — Anthropic, OpenAI, Mistral — have already run the data pipeline. Training, cross-validation, feature engineering, and model deployment are done. What remains is prompt and context engineering to modify model behavior, distributed systems design for multi-agent architectures (supervisor agents spawning sub-agents across heterogeneous infrastructure), human annotation workflows to label agent traces, and functional evaluation that goes far beyond precision, recall, and F1. Hetzel contends that data scientists habituated to binary classification metrics tend to under-evaluate the broader functional performance surface of agentic systems — whether the agent actually accomplishes the user's goal end-to-end, not just whether a classification head is calibrated.

He frames both sides of the debate formally. For data scientists owning agents: deep knowledge of neural network mechanics gives better appreciation of LLM failure modes, rigorous production deployment processes, and strong testing mindsets. Against: the model is already built; no training pipeline exists to manage; traditional ML metrics are insufficient for agent evaluation; and data scientists may lack proximity to the functional problem the agent solves.

The affirmative case for non-data-scientists owning agents: product engineers already treat LLMs as APIs — fetching structured data from an endpoint is core to their workflow. Complex agent topologies are fundamentally distributed systems problems more aligned with systems/application engineering backgrounds. Non-technical subject matter experts and product managers should directly author and iterate on prompts and context because they have the closest proximity to user intent.

Hetzel's recommended team composition is explicitly diverse: data scientists providing guardrails, statistical rigor, LLM-as-judge calibration (validating judge prompts against labeled datasets using precision/recall), and fine-tuning open-source models when needed; product/application/systems engineers implementing agent infrastructure and eval pipelines; and non-technical domain experts driving human annotation and prompt engineering.

On tooling, he briefly describes Braintrust's platform: offline eval experimentation (for pre-production confidence) and production observability (for post-deployment confidence), a human labeling component, and an agent/prompt playground for non-technical users. He describes a feedback loop where production traces feed back into the offline eval dataset, with grounded labels used to check whether LLM-as-judge evals maintain alignment with human agreement over time. The talk closed with audience Q&A affirming that the key organizational anti-pattern is isolating agent development to a single team rather than treating it as a cross-functional product problem.