At QCon London 2026, Jeff Smith presented the mismatch between AI coding models and real-world codebases. While AI tools are helping developers generate code faster than ever, Smith argued that the models themselves are increasingly stale because they lack the repository-specific knowledge required to produce production-ready contributions.
The presenter described this gap as structural rather than temporary. Most coding models are trained on snapshots of public repositories that may be months old, and they rarely have access to an organization’s internal code. As a result, the models can generate syntactically correct code but often fail to follow the architectural constraints and conventions that govern individual repositories.
One trend highlighted in the talk is the rapid growth of AI-assisted contributions. Mentions of AI tools in pull requests across several large open source projects increased dramatically between 2022 and 2025. However, acceptance rates have moved in the opposite direction. Smith cited data showing that pull request acceptance dropped during the same period.
This pattern suggests that AI is increasing the volume of generated code but not necessarily improving the percentage of contributions that can be merged. According to Smith, the fundamental reason is that every repository has its own unwritten rules. These architectural constraints often live in the experience of senior engineers or in patterns embedded in a project’s commit history rather than in formal documentation.
The talk examined how these rules shape real development workflows. Smith described two broad categories of constraints found across repositories. Architectural rules define how the system itself is structured. These include requirements such as component registration patterns, dependency handling mechanisms, or cross-file version synchronization.
Procedural rules govern how code changes are introduced and reviewed. These include pull request conventions, testing expectations, and review workflows. Although experienced contributors quickly learn these patterns, generic coding models typically do not.
Smith noted that these rules are often enforced implicitly during code review rather than through automated tools. As a result, AI-generated code can appear correct while still violating repository constraints.
To address this problem, Smith proposed repository fingerprinting, systematically identifying and documenting the unique constraints of a codebase. The goal is to extract the implicit rules that developers already know and make them accessible to both humans and AI systems.
The presenter also argued that current benchmarks for coding models often fail to measure what actually matters in production environments. Generic tasks such as algorithmic problems or language syntax checks do not capture the repository-specific constraints that determine whether code can be merged.
Instead, organizations should evaluate AI coding tools based on their ability to respect the architectural constraints of their own codebases. According to Smith, teams that explicitly document and operationalize their repository rules will have a significant advantage as AI-generated code becomes more prevalent.
The talk concluded with the growing mismatch between AI models and real repositories is not primarily a tooling problem but a knowledge management problem. Engineering teams already possess the knowledge required to guide AI-assisted development. The challenge is making that knowledge explicit and integrating it into the systems that generate code. By surfacing the architectural rules embedded in their repositories, organizations can close the gap between generic AI models and the unique requirements of their software systems.
.jpg "Humanscale’s New K Lounge Chair Is the Ultimate Home Office Workstation")
